How to improve our source control process

Question

Background:

We are using agile project management and follow sprints to do development. We are using a single repository in bitbucket.org that uses mercurial. Repository is hosting 3-4 products. There are many projects with 4 solutions for each product. There are some product specific projects and there are some shared projects.

We cannot do changes in the folder structure sine we already have a team city server that is doing automated builds which works well. But I am open if its really critical to handle it.

Current Process
Our current process can be described as follows:

• during development, we use hgFlow start and finish features (develop branch)
• for releases, we start releases, make them stable, finish the releases (new release branch from develop)
• for hotfix we start hotfix (new hotfix from release branch), and finish it (merges to develop and default)

Problems
Here are some of the problems with this process:

• when we start release, there are 3 other product's code which is not ready to release, is also present in the release and when release is finished, goes to default which is unwanted.
• when we change shared projects in development, it affects open releases and we have to patch it ther
• when we change shared projects in release, it affects previously released products

there are so many other technical issues that we discovered and we immediately stopped using hgFlow, and right now doing manually folder diff in order to start/finish release and hotfixes.

Some of the options that I thought of

• Create single repository to host each product's source code except shared product
• Create single repository for each of the shared project
• Separately maintain HgFlow and Tags etc.

That is too much work, say for example, when I want to start release, I have to go to product and each affected shared project to start a new release. same is true when doing regular development, for a single feature, if I want to change product and shared project, I have to do create feature in all of the projects.

How to better handle it?

Update - 9 feb 2016 Our products are based on .net, we are using bitbucket for hosting, mercurial and source tree with HgFlow for development operations.

Answer 1

Set up a company-level nuget repository. (I'm assuming .NET from the terminology you're using, but there are equivalents for almost all language frameworks.) Either build one on your own server, or use a service like myget, or simply use your TeamCity server's built-in nuget feeds.

Deploy your common code in packages to that repository, as an entirely different build in TeamCity. So, assuming the simple scenario of solutions A and B with common code in project P -- project P gets removed from Solution A and B and moved into it's own Solution, which builds one or more nuget packages.

Those nuget packages are included as references in any project that needs them. Just like packages from a public nuget server, except you have to tell Visual Studio where your package server is.

Now you can manage your branches in a project-level Mercurial repo, independently. The process for releasing code into the common project is:

Repository P

• Start feature
• Make changes
• Finish feature
• Start release
• Increase version
• Build
• Deploy
• Finish release

Repository A

• Start feature
• Update package
• Finish feature
• Start release
• Increase version
• Build
• Deploy
• Finish release

(For the record, you appear to be largely using hgflow, even if you're not using the command-line shortcuts, so I'm using that terminology here.)

Does that seem like a lot of steps? It is. But they're all tiny steps. Once you've done it for a while, it becomes kind of second nature.

And, most importantly, you'll notice that you haven't yet affected solution B. That's good. Your biggest problem has been solved. You colleague can continue to develop solution B without accidentally drawing your changes in. When one of you is ready, you can upgrade the package in solution B.

If you use beta releases (eg. 4.0.1-beta01) for untested code, your colleague won't accidentally pull in a change that you haven't tested yet (or as we like to call it, a bug).

This process is not without cost though.

In particular, you have to be careful of nuget dependencies. The more you have, the more heavy-weight the process to make changes to the bottom-level project. Better to have more projects or bigger projects than more dependencies.