Revision control concepts

The hardware engineering discipline has come a long way over the last half-century. Gone are the days of taping out a PCB design and keeping designs in a physical flat-file cabinet. New solutions throughout the decades required IT staff to create and manage network locations to store designs. Still, the maintenance and upkeep of the designs were on the honors system, and each design was only as robust as the weakest member of the team. Even if files had an ultimate destination, they were often passed around on “sneakernet” with code reaching systems on floppy disks or thumb drives. Many engineers used informal and ad-hoc release processes where they would zip their files and tack on revision numbers whenever they forgot something and needed to add it later.

Revision control has made huge strides in the past 15 years. File repos no longer need dedicated IT employees to set up and maintain servers. Numerous providers offer turnkey solutions for revision control. The most popular and functional tool has been the git.

Because git started as open source, anyone can create interoperable tools. Dozens of companies offer git platforms for you to keep your file changes. Although there are many different git solutions with their own features added on, they all have the same fundamental functionality and basic usage set.

Decentralized

Centralized revision control isn’t a single source of truth. It’s a single source of congestion. One of the biggest changes and advantages of git is that it is decentralized. Previous revision control software and strategies relied on a single source of truth. Centralization has certain advantages, like everyone having access to the exact same set of files and the final released files being unambiguous. There are a lot of downsides, though. In order to make a change to a file, you either use a system with the first change to a file going through, or you need to use a checkout/checkin process. Locking files can prohibit productivity and can often prevent people from contributing their changes while someone else works on the design. Centralized revision control also limits the types of changes you make to the design. It can be prohibitive to create an alternate design with prototype or experimental changes.

Decentralized revision control frees you to create sources of truth. Decentralized revision control gives everyone equal footing to look at and make changes to the design. Because everyone has a copy of the design on their local filesystem, they can make whatever changes they want to the files and push them to the server when it makes the most sense. Some might think that a centralized design would create a process that is more rigorous and produces the correct intended design, but with a decentralized approach, every change must be reviewed before being merged into the design.

Decentralization may seem backwards to some, but the added flexibility allows for more people to participate, more frequently.

Source files and output files

Output files are just as important as tracking source files. Modern revision control separates files into source files and output files. Source files are things like source code that compiles into an executable, which is an output file. Firmware is no different, with source files that get compiled into an output binary file that gets uploaded to the microcontroller, microprocessor, or FPGA. With hardware designs, the source files are schematics, PCB layouts, 3D CAD files, and cable wiring harnesses, to name a few.

Software and hardware track files differently. In a traditional software development git repo, the only files tracked are the human generated source files. The output files are not tracked, as the source code changes and are only attached or associated with the project upon release.

Revision control is subtly different from source control. There is a very strong argument in a hardware design process to keep many of these output files in the repo. Some people think the process is source control, and only track the source files. What’s really happening is revision control, where you track the revision of the files.

Tracking output files can improve your design. It may make a lot of sense to track output files. Bills of Material (BOM) can be checked-in to run planning activities like generating cost of goods sold (COGS) or finding availability of parts, or even replacements. Gerber files can be used by a mechanical engineer to determine if there is interference with fasteners, the enclosure, other PCBAs, or internal wiring. Fabrication packages can be circulated to contract manufacturers to obtain quotes and feedback into board stack-up, blind / hidden vias, or soldering processes that require parts to be moved. Saving netlists can allow people to run automated tests or checks against the design that would have to be manually run against a 2D schematic file.

Generated files aren’t always the same on regeneration. Checking-in the output files also guarantees that everyone has access to the same outputs. Two different users might have their ECAD tools setup differently and generate files in a different way, or with different features. There is also the matter of re-generating output files at a later date. With software, the argument is the files can always be re-created whenever you need them, but the same engineer generating the same files months apart might include unintended changes. The best time to capture the output files is the same time they are generated.

Hardware prototyping is different from software prototyping. A final argument for capturing output files is hardware prototypes differently than software. With software, you can create tests and run them, but with hardware you need to physically make the changes to a device. If you are capturing the changes as you go, you can follow along with what happened to the PCBAs in the lab as prototype firmware, white wires, and circuits attached with flying leads are added and tested.

Commit, Push, Branches

Commits are changes to files. With git, your changes to the design and files are captured in commits. Each time you make small and related changes to the file, you can create a commit with a message of what happened. The changes aren’t available to anyone besides those who have access to your local filesystem.

Commits are pushed to branches. Because git is decentralized, the committed changes must be pushed to the repo in order for anyone else to see or use them. Commits are pushed to branches. If you have a single branch system (which is not advised), all of your pushed commits will be to the main branch and considered “released”. If you use more than one branch, changes can be pushed to a development or feature branch and reviewed before being allowed in a main branch.

Branches organize changes. The use of branches allows segmentation of work into bugfixes, or the development of specific features. Different users might have their own branches so they can commit changes without a short-term conflict on files. Branches also allow users to run experiments or create proof of concept designs without messing up the main design files.

We’ll cover best practices for commits, pushing, and branches in later chapters, but for now all you need to know is that file changes are pushed to a branch.

Pull Requests / Design Reviews

Commits get merged from one branch to another using a design review (pull request). In git, the changes accumulated in a branch can only be merged into other branches using a pull request (PR). For example, if you created a branch to eliminate noise from an amplifier circuit and pushed all of your changes to a dev branch, you would need to create a PR and approve it for the files to get merged into the main branch.

At AllSpice, because we focus on hardware development, we call our PRs Design Reviews. We feel that PR is a confusing label as what is happening isn’t just a request to pull the changes from the branch, but to review the changes in a formal process like a hardware design review. Most electrical engineers are very familiar with a design review process and understand that it is more than an attempt to merge files. Design Reviews are an opportunity to make sure the intended changes work as designed and have no other side-effects.

Design reviews in git allow users to comment, request changes, or block the process. Once a design review is started on a branch, new commits can keep coming in to fix requested changes until the design review is completed and the files are merged. After the design review is approved, any new commits to that branch must be merged using a new design review. Some git processes ask or require that you delete a branch after merging and create a new branch for future changes. We do not recommend this for a hardware design and will cover more in the chapter on branching.

Design reviews are for reviewing. You should invite all stakeholders to a design review. In legacy hardware development, design reviews can take several days of meetings towards the end of the design calendar. By breaking up the changes into smaller chunks and reviewing them sooner, the reviewers can make sure their change requests have more impact on the project. Instead of carrying on for weeks with hardware that needs to change, your team members can digest the changes and make their needs known before it costs a lot of money and time to change.

Invite your purchasers. Planners and purchasers need to be added to design reviews. Hardware designs should generate an accurate BOM and COGs as soon as possible, even if the design is only fractionally complete. Cost overruns are one of the most common design problems, and catching expensive components or circuits early will give your team time to meet cost objectives.

A product is not a business, invite your marketers. Marketers should also be kept in the loop and on design reviews. One of the other common frictions in hardware development is marketing requirements not being met. Whether it’s cost per channel, output current, measurement fidelity or speed, UX features, radio output, or heat generated, someone should be reviewing the marketing specifications in addition to the team’s design specifications. It’s much easier to determine that you need to design a different radio if the marketing department sees that the range on the amplifier isn’t what’s been decided to be built.

Mechanical review with git. Mechanical engineers are a no-brainer to include on changes, being able to ensure board outline, part fencing and keep-outs are met, components aren’t too tall, and heat is properly dissipated.

Reviews encourage reuse. Experienced engineers on other projects can also be added to reviews. Design reuse is important and an extra set of eyes might notice that the designer is reinventing the wheel, when there is already an equivalent subcircuit on an older design that can be copied. When new components are added to the library, engineers can review them and recommend other components in order to not pull new part numbers.

Get your design for manufacturing feedback. Manufacturing engineers can identify things like single or low quantity parts that can cause additional costs to the design, when a similar part can be reused. Performing design reviews early and often allow the PCB traces to be evaluated early and offer views into whether the board will need to grow layers, or add expensive and slow complexities like buried and blind vias.

Develop tests before the product is released. Test engineers are often invited to contribute very late in the development, but adding them to all the design reviews allows them to specify test points and the addition of built-in self test (BIST) designs. It’s easier to write test firmware in lock-step with the design, than at the end when everyone must perform manual tests until the test code is completed. The test engineer can even work independently from the production firmware developer and add a separate firmware build for use in validation or verification.

Merging

Hardware merging is hard. One of the caveats of using git for hardware is that merging files has an added complication. Unlike software, hardware source files are often binary blobs. Schematics are 2D pictures of circuits that are stored as non-human readable text. With software, individual functional components would have their own files separate from their connections, so it is easy to make changes to a specific functionality without affecting the operation of any other component. The way most ECAD programs structure their designs, it is hard for multiple people to make changes to a design without stepping on each-other.

Hardware merging is manual. It is possible to open up some designs in a text editor or script, but it isn’t feasible for more than a fraction of the design to be done this way. When two different people make edits to the same schematic or layout file, it can be extremely difficult to reconcile the differences. Very often if two different subcircuits are on the same file, the easiest way to merge changes is to take one user’s changes via git, and then manually edit the files in the ECAD program. The future will give us more adept ECAD tools, but for now, if more than one person edits one file, one of their changes will have to be copied and pasted using a graphical editor.

Plan your merges or your merges will plan you. We recommend tackling your merging strategy before the work is done. It is much easier for someone to check out incremental work and make their additions, knowing there is no more work to be done, than to have someone manually copying and pasting changes. It is possible to lock files using git large file system (LFS), but there are many other ways like project management software or plain old communication to see who should or shouldn’t be working on a file.

Releases

Tags point to a revision. Releases are tags with files attached. Releases aren’t part of git proper, but are an addition that have been adopted by many tool providers. Tags are a formal part of git and are used to tag specific commits of code. Releases expand this by adding the ability to add files to the tagged design.

Deliverables to contract manufacturers are stored in releases. Although we recommend adding many of your output files to your tracked files that get committed to the project, releases are an opportunity to mark production or prototype ready files. It may take several commits to fix up your design once you think you are finished. It may take a couple of days of small changes to clean up silkscreens, move reference designators, or update fab files based on your final PCB stack-up or specification. Releases are a special way of marking the final files that other people will refer to when manufacturing your device.

Releases are PLM-light. If you use a separate PLM or ERP system, releases are where to store the files before they get uploaded to your planning or purchasing system. It’s up to your team whether you want to use the PLM as the final source of truth, and only have your users pull files from the PLM system. It may also be advantageous to give read-only accounts to your non-design team in order for them to do their jobs more efficiently.

Summary

Commit your changes, push to branches, merge with design reviews, and release files. Git has created a lot of value for companies, allowing them to finish their designs on a faster schedule, fewer bugs, and higher yields. There are many complexities to git, and it earns the reputation for having a steep learning curve, but it is relatively easy to get started. Simply put, git is a decentralized version control system for your source and output files. You push your committed file changes to a branch in the repository. After you conduct a design review, the changes are merged from one branch to another. When you are ready, you tag commits to give them a label. If you are ready to fabricate your design, or want specific files to be sent out, you create a release from a tag and attach your files.