D1->D2 part 6: fitting the final pieces together

A Tech Talk series about our migration from D1 to D2 (D programming language)

To actually get ported sourced to compile with DMD2, one last bit of work was needed. It was about two changes that felt very annoying to address in the usual “work for both D1 and D2” style, because of how common they are in D sources:

  • change of const decl = init manifest constants to enum decl = init
  • adjustment of this in structs, which changed from being pointer type to value type

Fortunately, shortly before this, I was experimenting with awesome Brian Schott’s libdparse and it occurred to me that grammar at current state is already compatible with D2, and that remaining changes can be easily automated.

After a quick word of advice from Brian, the d1to2fix tool was created, which did exactly that—lexing of intermediate sources and replacement of problematic token sequences before writing it back to the file. With that conversion, the porting process can be considered finished, assuming all earlier changes were correct, of course. Good thing everyone writes a lot of unittests, right?!

After adding more trials and experiments, eventually a relatively well-defined three-stage process was formalized to do porting of any application / library:

1.    Clean all the warnings reported by -v2 flag of DMD1 compiler
2.1    Do conversion using d1to2fix
2.2    Try adding const correctness until it compiles with DMD2
2.3    Revert d1to2fix changes and apply diff from step (3) to base sources
3.1    Fix tests failing with DMD2
3.2    Update Continious Integration jobs to run tests both with DMD1 and DMD2 (running d1to2fix in between)
3.3    Add Continious Integration job to automatically push converted and tested sources to dedicated git branch

In practice, Stages 1 and 2 are done in smaller chunks, of course, with as few packages touched in one step as possible.

This is much more work than plain, straightforward conversion to D2—but it enables crucial flexibility. Some applications can be switched to D2 earlier while more critical ones do the migration more carefully – but those will use the same library state despite the language difference, with no manual synchronisation of new features / bug fixes added. Regression control in Continious Integration jobs ensures that once some library is ported, it will stay in that special “D1 but compatible with D2” state, because otherwise feature pull requests simply won’t be merged.

And it is not a fatal commitment—if, at any point, the ported application encounters issues when compiled with DMD2, it can be reverted to D1 in a matter of minutes, allowing for uninterrupted business service while the matter gets investigated. It is planned that all daily development will continue on D1 sources until all projects are ported into compatibility mode and confirmed to work in production.

At this point it probably should be more clear why it wasn’t possible to use existing https://github.com/SiegeLord/Tango-D2 project: it is pure D2 code and adjusting it to compatibility requirements is a non-trivial effort. It seemed more practical to apply the same porting process as planned for other libraries and use existing D2 port as a reference when encountering any “tricky” parts; in fact, it was extremely useful in that role.

Looking into the future

It is very hard to estimate how long full transition will take—”not soon” is probably the best estimate under these circumstances. Right now we have our own version of Tango ported and another big core library about half way completed, all done by small group of people. The process is likely to speed up a lot once we get to actual applications, but there will also be dragons. Imagine testing a real-time application with new runtime and ported garbage collector and you may see why I am not calling hard deadlines to anyone.

At the same time, there are some things from the existing D2 ecosystem that may become used much sooner. I have already mentioned awesome libdparse by Brian—there are many tools based on it that should help a lot during daily development, primarily code style control and automatic refactoring tools. Phobos is also an important source of inspiration for updating designs of in-house libraries, even if it is not directly used.

All in all, I am totally looking forward it.