20 years of web development, or what has really changed in a programmer's work
When I started building websites, my biggest challenge was understanding what actually happens after you type an address into the browser. Today you can spin up a complete project with a database, an asynchronous job queue and a fully functional admin panel in a few minutes.
Does that mean building applications has finally become easy?
Looking back at more than two decades of web development, I see a completely different pattern. The difficulty never went away. It just keeps moving. Every time we got a tool that solved one annoying problem, we gained room to build much bigger things. At the same time we immediately ran into new barriers that nobody had even noticed before, simply because the scale was smaller.
Prehistory, or first understand, then get it running
When I started coding, PHP 4 was already well established, although you could still easily find live projects built on PHP 3. The standard toolkit for putting anything on the web was defined by an almost holy trinity, the Apache server, the PHP interpreter and MySQL 4.
In many projects of that era, scripts written in Perl or Bash ran in the background right next to the PHP code. If you could handle them well, people around you automatically saw you as some kind of real hacker. There were also brave souls who successfully ran entire dynamic sites in Perl alone, or even compiled pages in C++. Both were possible directly through the archaic CGI interface. To a self-taught beginner, this world looked complex, almost impossible to grasp.
The first serious barrier to entry had nothing to do with the syntax of the language, writing functions or loops. The real challenge was understanding what a server even is and where exactly the line runs between what the browser renders and what runs somewhere far away on the internet. You had to get it into your head that some files (HTML, images, client scripts) are downloaded straight to the user’s computer, the logic is processed by the server, and the database is yet another separate thing running somewhere alongside. And the best part was how you learned back then. I got my first documentation for Apache, PHP and MySQL, including simple examples, on a CD. Yes, really.
Everyone knows, though, that the best way to learn is on a living system. So the next milestone was setting up my own development environment, meaning a web server, a PHP interpreter and a relational database. Especially on Windows, just getting to the point where the browser displayed a trivial “Hello world” without errors was not a trivial task at the time. When you finally got through that installation ordeal and built a simple project that ran locally without a hitch, the next wall appeared right away, because you had to recreate the exact same setup on the production server, in a completely different, natural environment, which was Linux.
Deployment itself, the thing we now hide under the elegant umbrella of automated CI/CD pipelines, simply meant an FTP client back then and tediously copying changed files to the right directory. Absolute paths on the server, error log files, correct redirect rules. You had to take care of all of it by hand, keeping the small things in mind. Should the address include the “www” prefix? How do you generate internal links so they work both locally and after publishing to the production server? Is Apache definitely parsing .php files instead of cheekily serving their source code to the user as plain text? And what the hell is going on with file permissions?
Mundane details like these could eat up hours you would rather spend on the actual logic of the site. At that stage a programmer had to do everything, design and hammer out the interface, write the backend code, set up the system environment from scratch and deploy the project to production alone. The biggest difficulty was understanding and getting all the pieces of this fledgling puzzle to run.
Ready-made building blocks, or less writing from scratch
Over the years, as the code itself evolved, more and more libraries, ready-made CMS platforms and early frameworks appeared. Mambo (which later evolved into Joomla!), the ever stronger WordPress, lightweight CodeIgniter, and later mature tools like Symfony and Laravel changed where everyday work even started. We finally stopped rewriting form handling, routing, user authentication or the database access layer every single time.
That was the first big step forward. If a project needed classic content management, you could just take one of the widely available ready-made solutions and adapt it. You needed a custom business application written from scratch? Great, a mature framework took a load of boring, repetitive work off the programmer’s shoulders. Input validation, secure authorization, talking to the database or the caching layer no longer required writing yet another homegrown and often leaky implementation. You could reach for a tested library, so more and more of your energy could go into what mattered most, the specific business feature the end user actually needed.
But a ready-made block wasn’t free in terms of mental effort either. You first had to get to know it well and figure out how to configure it for your scenario, what its limitations were, whether it needed an extra extension compiled into PHP, and what to do when its error messages suddenly had to work in several languages.
Quite often, trying to force a ready-made solution into place exposed design flaws in the application itself. Writing typical, repetitive parts became much faster, but the difficulty moved to choosing, understanding and integrating the ready-made pieces well. Although this saved a lot of time, deployment itself still very often meant manual configuration on the target machine. Automatically updating third-party code or tracking the dependency tree remained a completely unsolved problem, and programmers had to look after backward compatibility on their own.
Dependencies, or “someone has already done this somewhere”
If you decided to use ready-made solutions back then, there weren’t many organized ways to get them. The simplest and most popular method was simply borrowing code from a trusted source or straight from a fellow programmer you knew. In its own way this was convenient, because it saved time and effectively quieted your conscience about security or responsibility. Code sent over by a good friend must be solid and secure, right?
In practice it was as simple as it could be. You created a separate subdirectory in the project, dropped someone else’s files there or unpacked a downloaded ZIP archive straight into it, and quietly hoped it would all work together. All that was left was fixing the paths in include or require by hand, and voilà, in the vast majority of cases it somehow worked.
The real headache started when you had to think about any kind of update. How on earth were you supposed to know that the library’s author had just posted a new, fixed version on a personal blog, or that a critical security hole had been found in the code? Back then hardly anyone worried about it. If something didn’t work exactly the way you assumed in your project, you usually had no qualms about opening that third-party file in your editor and making changes directly in someone else’s code.
Over time, sites like Phpclasses.org and SourceForge took over the role of shared repositories, becoming the community hubs of the time where you could dig up useful code for most purposes. The first official, systematic attempt to bring order to this chaos, modeled directly on Perl’s CPAN, was PEAR (PHP Extension and Application Repository). It worked nicely, but the problem with PEAR was that packages installed through it landed globally in the whole operating system (which often required root privileges), forced a rigid directory structure, and the packages themselves were eventually abandoned, never updated, and caused plenty of hard to resolve version conflicts.
Still nobody worried about updates and security, because who really followed the RSS feeds of those sites every day or regularly read industry mailing lists with vulnerability bulletins? Projects were ruled by an unwritten iron law of “if it works, don’t touch it”. Applications and sites were practically never updated after deployment, unless the hosting administrator decided to bump the PHP version on the whole machine (which often took down some of the hosted sites) or it turned out that someone had broken into the server and added malicious spam links to the site footer.
So in those days security and planned updates were almost completely ignored. Did that make projects less complex? No, although nobody said it out loud. Without proper engineering tools, the complexity was simply swept under the rug, which for many years effectively kept PHP out of many serious enterprise uses.
The breakthrough only came with the first successful attempts at standardization, namely the PSR recommendations, native support for namespaces and the debut of Composer itself. Composer solved the problem of automatically building the dependency tree, fetching the right library versions and managing updates at the level of the application’s own code. Today nobody can imagine daily work without Composer, and in production environments constantly keeping packages up to date and compatible with new versions is a basic part of the engineering craft.
Composer meant that adding another external library and reproducing its version in a completely different environment stopped being a complicated project of its own. The technical difficulty dropped so much that dependency management became standard in practically every project.
But along with this solution a new problem appeared right away, one of decision making. It’s harder to judge whether it’s worth making your application’s stability depend on yet another external package at all. Some repositories lose their authors’ support after a few months, it’s easier to accidentally pull in code from an unverified source, and the team has to constantly monitor the whole, often quite large, dependency tree, making sure a small change in one of the nested packages doesn’t bring down the entire system. We gained painless updates, uniform standards and a lot of saved time, but in return we took full responsibility for a lot of code we never wrote ourselves and which can quickly turn into technical debt.
Bigger projects, bigger problems
Since adding new features had become so easy and fast, applications started to grow quickly, and their complexity grew with them.
On one hand, systems naturally evolved into heavy monoliths that tried to handle everything, everywhere. In response, implementations of ever newer design patterns, best practices, thick handbooks and conference talks sprang up like mushrooms after rain. In reality things varied a lot. A good share of developers, true to human nature, proved immune to any architectural novelty, clinging to habits and patterns they had worked out years earlier. What’s more, years later some of the widely promoted patterns of the time were officially labeled programming antipatterns. Interestingly, the arguments behind rejecting them turned out to be perfectly logical and obvious, but in the heat of the earlier fashion nobody seemed to bother.
On top of that came another aspect. Besides the code itself, developed better or worse, the architecture picked up a whole lot of hard dependencies on external services and system daemons that were needed to deliver modern features properly. A few examples below.
- Full text search based on Apache Lucene, and in later years advanced Elasticsearch clusters,
- All kinds of asynchronous job queues for handling background processes,
- Multi-level caching, not only of data in memory (like Memcached or Redis) but also of the compiled intermediate code itself,
- Ever more extensive integrations with external services, APIs and webhooks in all sorts of data formats,
- Complicated online payment processing,
- Alternative ways of persisting data, entire analytics and statistics subsystems, and much more.
Over time, even a code layout that had been very well thought out at the start stopped matching the real boundaries of the fast growing system.
In projects like these, a single change rarely ended with adding just a few lines. A developer had to know upfront which other modules depended on the part being touched, how to reliably test external integrations and what would actually happen during the production deployment. The eternal promise that “we’ll rebuild it someday” almost always lost against a hard business deadline for shipping the next feature. That’s how good old grep had to give way to advanced search in IDEs and ever more sophisticated static analysis tools.
Systems like these grew without restraint, dragging along a huge amount of complexity in their runtime environment. It had long stopped being just an Apache daemon wired to a PHP module and a single relational database. We were talking about a set of at least several independent services that required every programmer on the team to have at least basic configuration knowledge just to get any working development instance running.
That, of course, immediately created another problem, because every programmer worked on a slightly differently configured machine. The chaos became especially visible when automated tests became common in projects. The “it passes for everyone in the office but fails for me” scenario became a programmer’s daily bread. It meant more hours lost on debugging, only to discover in the end that one of the local services had a different memory limit, a differently configured InnoDB in MySQL, a different default time zone or a slightly different library version. A nightmare.
I remember commercial projects where onboarding a new hire and just preparing a working local environment for them could take several full working days. The difficulty was no longer in writing the code itself. Typing out lines was getting easier over time. It simply stopped being enough. For the code to work at all, you had to recreate the whole technological environment around it, without which the application didn’t run and which became more and more complicated over the years.
That’s how the difficulty of our work expanded from programming alone to staying consistent with all the existing infrastructure, and with it, to fighting for predictable behavior of the whole system as one coherent thing.
Vagrant and Docker, or the environment written into the project
That’s when virtualization entered the game. First through virtual machines managed by Vagrant, and soon after through the revolution brought by Docker itself, we finally managed to tame a good part of that infrastructure chaos. Instead of handing a new team member a multi-page, usually outdated list of instructions in the wiki, you could prepare the complete configuration of the whole environment just once and keep it right in the repository, side by side with the source code. With a bit of luck and a properly prepared docker-compose.yml file, you just cloned the project, typed one command in the terminal and a few minutes later you could start the actual work.
This really changed our daily routine. How much easier it was now to guarantee identical versions of databases, queues and interpreters for every programmer on the team! Bugs could be reproduced predictably, and automated tests finally started giving reliable results. At the same time, though, someone on the team had to build those images, configure the virtual networks connecting the containers, take care of mounting data volumes, keeping them consistently updated, and a safe deployment process to the servers. Docker by itself didn’t magically make every project run in a vacuum. It simply gave engineers standardized tools to finally make that process repeatable.
For many developers, joining a new project became easier. The difficulty, however, moved to preparing and maintaining that environment, blurring the traditional line and forcing close cooperation between the people writing the application and the people responsible for the infrastructure. What’s more, from then on you could fully version not only the application code in version control (Git), but also the declarative configuration of the entire operating system.
As you can guess, this solution brought a new wave of problems, pushing up the complexity of already complicated IT systems. Basic issues such as the ephemeral nature of containers and data persistence, meaning the painful discovery that removing a container also removes all the database tables stored in it, were usually mastered right after the first painful mishap. But with modern cloud platforms, which quickly became the natural environment for running containers at scale, new challenges appeared around security itself, permission management and orchestrating hundreds of interconnected processes.
For people just entering the world of programming, having to master a new class of tools became quite a barrier to entry. Everything was fine as long as it worked, but when something broke, debugging even the simplest error related to the local copy of the development infrastructure was a real problem for the “uninitiated”. Still, the balance of gains and losses remained indisputable, because virtualization gave us far more benefits than all the earlier, chaotic attempts to solve these problems. For the first time ever it also let us effectively separate that invisible, always problematic line between developing the code and the infrastructure it is supposed to run on, while keeping elegant isolation between local environments, staging and production.
Architecture, or “we don’t need that”
As systems kept growing and more and more engineers worked on a single product, the simple question of whether a feature works correctly was no longer enough. We had to ask where in the system’s structure a given piece of logic should live, what it may depend on and what it must not depend on, and whether it can be extended after a sudden change in business assumptions without rewriting a large part of the application.
For this kind of strategic analysis, help came from engineering approaches and practices that largely originated as general concepts back in the pre-internet era. Domain-Driven Design (DDD), Clean Architecture, hexagonal architecture and Event Sourcing turned out to be excellent weapons against complicated domain problems. Add to that techniques for modeling processes live, backed by Event Storming sessions with the business.
Of course none of these techniques was treated as a mandatory stage in the development of every site. A simple system or a classic business card website needs none of these elaborate approaches, and applying them thoughtlessly just in case can artificially pump up complexity to absurd levels. But if the application’s life cycle is long enough, the project keeps growing and a sizable team develops it, a lack of any architectural planning inevitably leads to paralyzing chaos. Then, sooner or later, the techniques above turn out to be simply necessary to keep the project maintainable at all.
Larger organizations naturally developed highly specialized roles and teams. Some programmers focus only on the interface layer (frontend), others dig into backend architecture, and still others take care of the cloud platform and infrastructure, automated quality testing (QA) or the direction of the product. Although the boundaries between these roles are flexible and fluid, they mean that some decisions are made by an architect and some by the whole team through discussion.
That doesn’t free an individual programmer from thinking about architecture, quite the opposite. The key is for everyone on the project to look at it in exactly the same, consistent way. It matters a lot in daily work. Operating within defined, rigid boundaries, a developer makes dozens of micro decisions every day about how to logically fit newly written code into the thicket of already running, critical parts of the existing ecosystem.
This gave us a real ability to build and reliably maintain much bigger, more ambitious digital products. In return, though, we had to master the hard art of making decisions together, drawing lines of responsibility and communicating constantly between people and distributed teams, in a way that is fully transparent and understandable to every member of the organization.
AI, or the difficulty moves once again
It doesn’t really matter whether we look at the frontend, the backend or server configuration, wherever those servers are. For twenty years, as an industry, we did everything we could to speed up our daily work or even automate every possible part of it. You can see it in versioning code changes, automated zero-downtime deployments to production, automatically pulled updates, and strict architectural rules that brought logical order and disciplined the development process.
For years we aimed for faster, more stable, more repeatable and cheaper. The point was to make running complex projects as predictable as possible, so that a sudden change of individual developers wouldn’t turn the whole software production process upside down.
Looking at it broadly, everything today has become incomparably easier. Wasn’t that the original goal of this whole technological race? Every day we draw freely on everything that has been worked out in recent years, universal code standards and PSR recommendations, the full history of decisions safely stored in repositories, mature architectural patterns, ready-made, well documented frameworks, dependency trees resolved automatically by package managers, scalable distributed systems and High Availability environments, and even declarative versioning of machine configuration and the entire infrastructure (IaC).
Has this made modern systems simpler? No. We simply stopped worrying about things that two decades ago required almost separate, pioneering engineering projects to solve. At the same time, though, the overall complexity of systems and the overall barrier to entry into advanced software engineering grew by orders of magnitude.
And that’s the dense ecosystem generative AI walked into.
Artificial intelligence can prepare a working draft of a new feature in a few seconds, generate a whole module, or even the outline of a complete application. It will readily propose a set of unit tests, explain the logic of someone else’s convoluted code you are seeing for the first time, or help you pick up the syntax of a brand new tool in no time. It’s a big productivity boost that speeds up work at every stage.
But does that mean that from now on, as if by magic, efficient, scalable and secure applications of any size will start building themselves?
In small hobby projects written for the drawer or in simple internal tools, generated code like this will often be more than enough. But with serious commercial products, someone has to take full responsibility for them running reliably and without errors, and I don’t mean only purely technical responsibility, but business and financial responsibility too. On top of that come plenty of laws, data security standards and market regulations that every commercial system has to meet just to operate legally.
Code generated by an assistant still has to be judged competently. You have to check whether it addresses a real business problem and whether it fits the project’s standards and architecture. You have to know how it will behave on an error, under heavy load, or when an external service suddenly changes its response format. And finally, whether it can be safely maintained and changed for years to come.
That last point seems the most critical to me, because building a product is really only the first stage of an application’s life cycle. It doesn’t really matter whether someone builds it at the best software house, uses freelancers from the other side of the world, or uses AI for it. Isn’t that right? And that’s where the bulk of all costs will be hidden.
The easier and faster it gets to generate more building blocks for an application, the more important it becomes to understand the architecture we’re adding them to.
For a language model it makes no difference at all whether it has to add a complex, running Elasticsearch cluster to your project. Given the right permissions, it will configure it without hesitation and happily deploy it even to production. In a couple of hours you’ll get a super scalable and, in theory, completely secure (because it will assure you it’s “production-ready best practice”) way of searching your data. The only question is whether your application needs such a solution at all, whether it isn’t using a sledgehammer to crack a nut at your data scale and, most importantly, who will pay for running a few more machines on AWS at the end of the month?
AI lowers the barrier to entry into programming itself, and a person taking their first steps will see a working, satisfying result on screen much faster. But getting a working prototype to run and understanding its long-term goals, architectural limitations and vulnerabilities are still two completely different worlds. It’s exactly the same pattern we as an industry already went through with the birth of ready-made frameworks, global library repositories and containers. The difference is that today everything happens much faster.
So what has really become easier?
Today’s applications look nothing like the ones we so eagerly put online ten or twenty years ago. Although in everyday talk we still use the same terms (database, server, HTTP request, form), the level of complexity, the density of interdependencies and the scale of modern software have grown exponentially. Who was dealing with sharding, for example, twenty-odd years ago? And until recently, this whole technological leap was only possible thanks to the coordinated, painstaking work of human teams.
After more than twenty years in the industry, thanks to generative AI we can build in a single day something that not long ago would have taken weeks of work. Granted, besides a bit of time it also takes a few extra dollars for tokens and model subscriptions. In return, though, we no longer have to write many basic, repetitive mechanisms ourselves or rebuild server configuration by hand every time. In fact, at times it can feel like we no longer have to program or configure anything from scratch in the traditional sense. Most bugs, bottlenecks and integration problems at every stage of work can be diagnosed quickly, and often even fixed automatically with a few well aimed prompts.
In this new balance of power, the programmer becomes a kind of “product engineer”, which shifts the center of gravity from producing code to modeling features, designing business logic and defining how the system works.
Will this make the applications themselves any simpler? I dare say quite the opposite. If we as humans could achieve this much technical progress without the help of artificial intelligence, today we can only imagine how complex the systems we’ll be building and maintaining in a few years will be.
Despite the loud announcements of visionaries and the impressive demos of the newest models, their hard limitations are still easy to see in daily work. The new tools are like brilliant super developers on steroids who can work for us 24/7 without getting tired. Their cognitive abilities, however, are not (or at least not today) unlimited.
You can see it clearly when working with larger modules, especially during complex, multi-step building of new functionality or refactoring. At some point the models can simply lose the thread, completely forget firm architectural decisions made a few hours earlier, and, without enough attention and critical thinking from the programmer, eagerly start implementing edge cases that can never actually occur in the given system.
Of course, just like any human developer joining a team, AI first has to get access to all the key artifacts and project decisions. And that brings more difficulties, because you have to decide what to do about credentials, how to document business logic and architectural assumptions clearly, how to share context in a team where machines take part in every stage of work, how to plan tests and fast feedback loops, and how to catch the subtle bugs the machine introduces into the logic of modules and the connections between them.
On top of that, artificial intelligence still doesn’t eliminate any of the tools we so painstakingly developed and brought into our work long before it existed. Modern tooling, static analysis, CI/CD systems, containerization and business metrics work in favor of models exactly as they do in favor of human programmers. These tools make the work easier and increase confidence that we’ll see a correct, stable result much faster, and cheaper too (for now). A deliberate choice of system architecture and of the proven engineering tools we forged over the last decades matters even more today than ever before.
A programmer’s work is changing a lot and gaining unprecedented momentum. It gives us the chance to create even better, more advanced and more interesting things. Today, much like shaping digital clay, we can turn ideas into useful digital products very quickly.
One question remains open. If we spend so little time actually programming today, are we still programmers?