They say, "Culture eats strategy for breakfast." A well-established culture truly is a powerful tool that helps to keep the way various challenges are addressed consistent, increases autonomy, and saves time in making decisions. I'm saying "well-established" because culture exists regardless of whether you put effort into shaping it or not. Same as if you don't have a dress code, people won't come to the office naked; if there is no guidance on company culture, they will contribute with their mindset to an overall company culture.

In small companies, alignment happens naturally. When everyone talks to everyone any mindset differences are surfaced and discussed early. When forming a team, people tend to find similar-minded peers. And if it happens that they don't share the same mindset and cannot align on it, they don't survive long together.

When a company grows, it becomes gradually harder to maintain a consistent mindset across loosely coupled parts of the bigger organization. In cases of hypergrowth, company culture can be considerably diluted as new joiners do not always have a chance to work with culture carriers and can even form their own culture in a newly created department.

To handle this problem, some kind of written culture manifestation is created. It is quite often called "values and principles". It may also be published in the shape of an onboarding material (e.g., Valve's Handbook for New Employees) and literally can be called "manifest" (e.g., Agile Manifesto - it is not a company's artifact, though it is a set of values and principles for particular software development culture).

Such a document aims to provide high-level guidance without detailed instructions for any possible scenario. It is also often publicly available and used as company identity material and advertising / setting expectations for the new hires. This gives an excellent opportunity to analyze and compare different formats, focus, and essences in how different companies define them.

The document usually (though not always) is very concise and follows a pyramid structure, starting with the highest abstraction level of "values" and building on top of that into more details, defining principles, or even specific rules. It is common to define either of those levels as a set of rules of precedence (smth. over smth.). The classic example from Agile Manifesto: "Value individuals and interactions over processes and tools".

Example of Lingualeo

I wanted to start with this example for two reasons:

1. This is the first company I've faced explicitly formulated culture

2. The way it was defined was very short, which makes it an easy start for such a blog post format

Lingualeo Values, in their first version, were defined as three different aspects of human interaction sorted in the priority order:

• Respect

• Trust

• Support

I remember there were more detailed documents describing it. But I don't have any at this point, so I'll do my best to explain those principles and their motivation in my own words.

Respect

Team members at Lingualeo base their interactions on the principle of mutual respect. We can disagree, but we care to express our disagreement in a respectful way. We also value each other's efforts in growing the product and extending its functionality.

Why does it matter? The team was built of very young and motivated people, so we quite often went into very heated discussions about things. Without a principle of mutual respect, those discussions could easily grow into conflicts and develop into long-term tension between team members, which would be a burden that would slow us down to complete paralysis.

Trust

We trust each other in the way we do work and make decisions.

Why does it matter? Trust is a foundation of delegation and autonomy. If you can't trust your peer, you would have constant overhead to cross-check and doubt the decisions before their impact can be measured.

Support

We support each other and help to achieve goals by resolving dependencies in a timely manner.

Why does it matter? Even in the early days, we were 2-3 teams. Each of the teams took care of the specific area of the product. Sharing a common codebase, we had dependencies, and without being able to resolve them, we couldn't advance in our own goals.

Summary

This is the most compact value system (company culture definition) I have ever worked with, and of course, the way we worked was not defined only by these three principles. But I like the fact that being short, it focuses on human aspects that can have a significant influence on the way people interact within an organization and the achievements of the team. The culture remained mostly unchanged for the 3.5 years I worked at that company during its growth from 10 to about 100 people, acquiring about 15 Million users worldwide.

In this post, I would like to share how PARA method can be used to structure team's knowledge and documentation. Although it can be used with any wiki platform, we use notion, so examples and screenshots will be from there.

What is "PARA" method

The PARA method is a method to organize digital life (knowledge, notes, etc.) in external storage (google drive, folders on your computer, note-keeping apps). It is authored by Tiago Forte and covered in his book "Building a Second Brain".

Essentially it suggests organizing your knowledge into four categories:

P - Projects. Short-term efforts in your work or life that you’re working on now.

A - Areas. Long-term responsibilities or ongoing activities with no set deadline.

R - Resources. This section holds the information you would like to keep and refer to in the future.

A - Archive. The category where you'll put items from the other three categories that have been completed or are no longer active.

How to apply PARA method to structure team space

Let's use PARA categories as sections of the Team's home in the org's wiki.

Projects

"Projects" section naturally fits as a root of your recent, current, and future projects. You can use different terminology and call them initiatives, roadmap items, or epics. Still, they should fit the classic project management definition: have a defined beginning and end, a specific scope of work, and a set of objectives to be achieved.

Projects can have a product or feature delivery as an objective or be internal ones aiming to improve processes within the team. For example, introducing DORA metrics or adopting a new tool in the team.

In this section, it makes sense to sort items in reverse chronological order (newest to oldest). Each subpage here is just a root holding references to relevant information, such as:

• Project meta information:

  • Project definition in the project database

  • Project metrics and success criteria

  • Project roles (i.e. Stakeholder, DACI / RACI, or another framework you use)

  • Links to other systems (Jira epic, Miro project, Figma folder, etc.)

• Ongoing project documentation:

  • Architecture decision records (ADRs)

  • Brainstorms or other workshops protocols

  • Meeting notes

• Feature or technical documentation:

  • User stories

  • Flow / sequence / ER or other kind of diagrams

  • Runbooks for engineers or support specialists

  • Tracking schema, service catalog, etc.

Organizing all relevant documentation under one root makes it easier to find it throughout the project's lifetime.

Areas

The "areas" section can hold projects or business domains which are not in active development but require maintenance. Not every piece of ongoing project documentation is moved here once product goes into maintenance, only some long-term ones, like features description, runbooks, key ADRs, etc., would be helpful here.

You don't move projects to areas once they are complete. You rather distill long-term documentation from projects to areas. In fact, areas can have a broader scope than a project. For example, an area can be "paid subscription management", but a project belonging to this area can be "introducing a new payment provider X".

In addition to product domains, "areas" section can represent your engineering practices. If you perform post-mortems as part of your incident response process, it's a good idea to put them here. Your engineering or quality strategy, your productivity practices, and your team-building events - can all continuously be tracked in subpages of this section.

These subpages are better to be sorted alphabetically, or from most used to rarely used. If there are too many of them, it may make sense to introduce logical groups.

Resources

This section is perfect for the information you want to keep up to date for reference:

• Team's mission and roadmap.

• Working agreements (i.e. definition of "ready" and "done", meetings structure, core working hours, kitchen duty schedule, etc.).

• Contact book of your teammates.

• Rituals guidelines.

• Templates for communication.

• Team logo

This list can go, but from these examples you can see what all that items have in common: you are only interested in a current version of them. They also support your daily work in some way.

Archive

The number of items in other sections can grow and become hard to navigate over time. It is important to keep it clean and have only relevant information at the reach distance of your hand. Projects are finished, features are being sunset, working agreements get deprecated, and new versions of strategy are written.

It is a good habit to review elements linked from the main page regularly (i.e. quarterly) and move them to the archive. I would recommend having the same sections in the archive to make it easier to find archived elements by type. The archive will become hard to navigate soon too, so I would recommend organizing elements in each section by years (by the moment it was moved to the archive).

Can I add something else to the page?

Of course, these four sections are not set in stone. You may want to have some information available deeper in the structure to be duplicated on the main page.

I'd consider putting important information in sight:

• team mission, values, moto

• roadmap snapshot

• members list

Being there, this information will catch an eye and serve as an "information radiator" making it readily available to everyone (even if you don't look for it). Just be careful and make sure you keep it concise and do not overwhelm the page.

Notion templates

To demonstrate the idea, I've prepared some notion templates. Feel free to clone them to your notion space and try to organize your pages with this approach.

You don't even need to commit and move your existing pages. You can link them (aka "page mention" in notion) inside the new structure to get the feeling of whether it works for you.

• Santa Crew - team's page template [Wide]

• Santa Crew - team's page template [Narrow]

• Santa Crew - team's page template [Collapsable]

There are more than enough articles about best code review practices, but I'd like to summarize my thoughts and experience in this post.

Define "code review"

According to Google's code review developer guide, "A code review is a process where someone other than the author(s) of a piece of code examines that code."

I would also add to that definition some goal setting:

Code review is a collaborative act between the author and reviewer aimed at delivering the code increment in the best quality and most suitable for further maintenance at a given time budget.

The main output of code review is the improved code quality of the particular change, highlights of issues the author(s) of the change could have missed, and alignment on the balance of trade-offs made in the increment.

Before formulating review principles, let's discuss the requirements for the review request itself:

Creating a PR

1. Keep Pull Requests small (ideally under 400 lines), and prefer several small PRs over one big (with sensible exceptions, i.e., for refactorings, automated code changes, etc.). Small PRs are faster to review, and reviewers are more likely to find issues (see "LGTM syndrome").

2. Separate refactoring and actual business logic change into different PRs. Considering that refactoring can be extensive and unrelated to the business logic change, having both in one PR will make it harder to focus on each.

3. Before requesting the review, make sure all automated checks pass (linters, static analysis tools, automated tests, etc.).

4. Before requesting the review, look through the pull request yourself for obvious improvements (leftover debug code, accidental changes, etc.).

5. Format your pull request's title and description according to guidelines, don't leave an empty PR description template.

6. Add comments explaining certain decisions in the code or providing context (link to documentation, discussion happened elsewhere, etc.)

Communication style in code reviews

1. Always be respectful, friendly, and professional to your co-workers, and preserve a positive and constructive communication style.

2. Keep your comments concise. If more than one paragraph of text is needed, leave a note but communicate details verbally (in a shot call or over the desk).

3. Be explicit about whether the change you suggest is strongly needed or nice to have.

4. Provide reasons for changes with possible examples, links to guidelines, or best practices when requesting a change.

Code review principles

1. Technical facts, company dev guidelines, and business requirements supersede personal opinions and preferences.

2. Respect the task scope. The "boy scout rule" is great and shall be used when there is an opportunity, but be mindful about not creating a significant delay to the task delivery. If necessary, create follow-up tasks for addressing valuable comments that can't be fixed now.

3. Allow a variety of solutions for edge cases. To support the pace, consider handling missing cases with graceful errors. Iterate within a task or create follow-up tasks if necessary.

4. Done is better than perfect. Delivered code has more value than infinitely polished.

Roles of team members and outside reviewers

It may happen that a person from outside of the team can add their review to the code even when not requested explicitly. Generally, it is a great practice providing transparency, knowledge exchange, and innovation.

Everyone is free to drop by the code review and add comments, but only the team members make the decision about merging.

This principle is crucial for supporting the culture of team autonomy, ownership, and a spirit of healthy compromise. Outside reviewers often don't observe other factors in balancing trade-offs, so they cannot make an informed decision about the approval.

What code review is not

It is also essential to understand what code review isn't.

1. Code review is not an attempt to make the code absolutely perfect. Without a limiting factor to balance participants' strive for perfection, review and iterative fixes can take forever, reducing the value of such increment.

2. Code review is not a competition of who can write more comments on the code change or who knows more ways to implement something. It may feel natural to write many comments and get into the "I'd rather do it in another way" trap. But it is crucial to understand if implementing the comment improves the code.

3. Code review is not the act of gatekeeping the delivery by some authoritative person or a group. This approach, first of all, doesn't scale well. But also contradicts ownership and autonomy principles while possibly also working against innovation.

A small disclaimer

The principles formulated above are crafted for commercial product development. Some of them may be debatable in the case of open-source development, where effort and time to market are less pressuring (which is also debatable) factors.

• Deploy nginx server as an additional service to handle redirects.

• Handle redirects at your ingress controller.

The first one is safer but a bit more complex, the second one utilizes the ingress controller of your cluster, so you get redirects "for free", but it comes with a penalty. Let's look at both of these ways in detail.

As an example, we will set up redirects from old-domain.tld to new-domain.tld for two paths:

• /pages/about → /about with 301 Moved Permanently status

• / → / with 302 Found status

Additional service to handle redirects

Not to create all k8s resources from scratch one can use Bitmani's nginx helm chart.

Here is an example of values.yaml file that configures ingress to route the traffic to an old domain to the nginx container and injects a custom "server" block to implement redirect rules:

ingress:
  enabled: true
  hostname: old-domain.tld
  ingressClassName: nginx
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
  tls: true

serverBlock: |-
  server {
    listen 0.0.0.0:8080;
    location /pages/about {
      return 301 https://new-domain.tld/about;
    }    
    location / {
      return 302 https://new-domain.tld/;
    }
  }

To install the chart one should run the following commands (assuming the file above is located in the current directory and named my-values.yaml):

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install redirector bitnami/nginx -f my-values.yaml

In this case, the ingress controller will proxy requests to the "redirector" instance of nginx and it will respond with appropriate redirect headers.

Ingress configuration to handle redirects

But why use an additional nginx container if you probably already have one in your cluster — your ingress controller? Is it possible to inject redirect configuration directly into it? The answer is yes, but there are several issues.

The only thing you need is the ingress resource looking something like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-redirector
  annotations:
    nginx.org/server-snippets: |    
      location /pages/about {
        return 301 https://new-domain.tld/about;
      }    
      location / {
        return 302 https://new-domain.tld/;
      }
spec:
  ingressClassName: nginx
  rules:
    - host: "old-domain.com"
  tls:
    - hosts:
        - "old-domain.com"
      secretName: old-domain-cert

As you can see, the nginx configuration is injected via nginx.org/server-snippets annotation and follows nginx configuration syntax. Otherwise, ingress doesn't look any different from a regular one, except there is no need for backend service configuration as all the traffic will be handled by the ingress controller itself.

The "issues" I mentioned are:

• Your ingress controller has to be based on nginx (maybe other controllers also can be configured for the task but they are not covered in this blog post).

• Snippets configuration should be enabled on the controller level. Either with enable-snippets command-line argument or, if you use helm to deploy your ingress controller, by setting controller.enableSnippets parameter to true in your values.yaml file.

• Third, it introduces a certain risk: if the configuration snippet has invalid syntax it will block configuration refresh for all the services using this ingress controller in the cluster. Even other ingress configuration changes won't apply. See more in "Disadvantages of Using Snippets".

To reduce the risk of invalid configuration, you can use a public helm chart I've created: https://github.com/obukhov/ingress-redirector. Of course, it is still possible to break it with some special character in "from", "to" or "code" parameters. But at least it is harder to do it unintentionally and values.yaml file will look a bit more friendly:

ingress:
  hosts:
    - host: old-domain.tld

  tls:
    - secretName: old-domain-tld-tls
      hosts:
        - old-domain.tld

redirectRules:
  - from: /pages/about
    to: https://new-domain.tld/about
    code: 301
  - from: /
    to: https://new-domain.tld/
    code: 302

Summary

What solution to choose depends on your situation. Do you have several development teams, and such implicit dependencies between their services in a shared runtime can cause an issue? Or is it just your pet project, and you really would like to avoid deploying an extra nginx container that will require resources? Kubernetes can serve your needs anyway.

This post intends to be a short cheat sheet for those who provide peer feedback (respondents). It targets people who are about to write their first peer feedback and don’t know where to start, as well as those who have done it many times and need a refresher to keep the quality high. It focuses on written regular feedback. However, some advice can still be helpful for verbal or ad-hoc feedback.

Why is it essential to give feedback

Most professionals want to become a better version of themselves. It’s hard to do relying only on self-reflection. Planning for improvement based only on your own observations is similar to Münchhausen pulling himself out of a mire by his own hair. The blind spots are nearly impossible to get rid of without external input.

Why should I care about delivering feedback properly

Properly delivered feedback has a higher chance of converting into behavior change. And this is better for several reasons:

• Interaction with the person will become more productive and satisfying for you and other teammates and for the person themself.

• You will see a higher return on your time investments.

• The most valuable gifts are the ones money can’t buy, or in other words, something the person cannot obtain themselves. Feedback is this kind of gift: giving feedback that helps others grow makes you a better professional as well.

Prepare to give feedback

As we write peer feedback only occasionally, it requires concentration and a mindset switch. Prepare the right environment for yourself to provide the best feedback you can:

• Allocate enough time. The time you may need can easily be over an hour, depending on how detailed your feedback is.

• Eliminate all distractions (turn off notifications, go to a meeting room or a booth specifically booked for it).

• Remember (or even write down) all recent interactions with the person you are giving feedback. Projects you’ve been working on together, discussions you had one-on-one or in a group, reviews, documentation, and informal conversations that stood out.

Don’t try to optimize by multitasking or speeding up. Rushing the feedback may be more harmful than not giving it at all.

Anatomy of peer feedback

The most common peer feedback structure is two contrasting blocks: what the person does well and what they can improve. It is often treated as “positive” and “negative” feedback. I would suggest thinking of those in classic retrospective categories: start, stop, continue. Where both “start” and “stop” are improvement suggestions, and “continue” is something the person does well.

Shaping your feedback

There are several practical rules for shaping the feedback to make it more convincing and impactful.

Differentiate observations and interpretations

Observations are objective facts anyone can agree on. In the context of feedback, it is, for example, something the person has said or done. Interpretations are subjective. It is often an opinion you’ve formed based on observation or an emotional response to the action.

Compare:

• “You don’t value the input of others in the conversation.”

• “You have interrupted another person in the conversation ...”

Caution: Avoid giving opinions not supported by observations and vice versa. You don’t give them a chance to reflect on the situation without providing any observations. If you miss the interpretation, it is unclear why changing or keeping the behavior is valuable.

Emphasize interpretations

Don’t state interpretations as facts. Word it in a way to emphasize that it is your interpretation. It makes feedback more relatable and disarms natural defensive reactions. In the end, you can’t disagree about facts, and influence on others is precisely what you want to know.

Compare:

• “You don’t value the input of others in the conversation.”

• “... it made me feel that you don’t value the input of others in the conversation.”

Caution: I mentioned before that you can’t disagree about facts. If you feel the person can disagree about the fact you’ve provided, look again. Probably your “fact” has a little interpretation in it. In this case, try to separate one from another once again.

Compare:

• “You have made a rude joke.”

• “You have made a joke which I consider not appropriate.”

Be specific about your observations

Even if you think a behavior you describe is typical for the person, it is crucial to provide specific examples. One or two most recent ones are usually enough. Once again, It gives a chance for the person to reflect on those moments as well. It makes your feedback much more convincing and helps the person to see their behavior in context.

Compare:

• “You always interrupt others in a discussion.”

• “You interrupted Mike in the refinement a week ago and Peter in the last retro.”

Caution: If you cannot recall recent examples of the behavior you wanted to mention, it may be worth self-reflecting. Does the person still demonstrates the behavior, or does your perception trick you? Can a previously formed impression stick with you for longer than needed?

Desirable behavior

The framework above is also known as “AID feedback.” AID stands for “action, impact, desirable behavior.” The last one can be pretty obvious yet worth mentioning; the first two are absolutely required.

Positive (“continue”) feedback

It seems like positive feedback is the easiest to provide, as it doesn’t imply any tension or bad feelings when receiving it. It is crucial to offer it and deliver it properly. Without acknowledging their positive behavior, the person will not know that others value it and can stop doing it. The following questions can help to come up with positive feedback if you struggle to find it:

• When the person exceeded your expectations or fulfilled them exactly?

• When was the last time you were thankful that you were working with the person?

• What are the recent achievements of the person and their team, and their contribution to those?

• What recent improvements have you noticed (either based on your previous feedback or not)?

Improvement feedback

Some people feel bad talking about improvements. They may feel they are complaining about others, afraid to hurt their feelings, or cause trouble. When “packaged” correctly, the feedback will not hurt, and ignoring possible improvements, can cause much more trouble in the future. To facilitate writing improvement feedback, think separately of “stop” and “start” improvement suggestions.

In general:

• Think of moments when you would do something differently if you were that person.

• Think of moments when your expectations were not met (i.e., dependencies or attitude).

Suggest stopping something:

• If behavior can hurt the motivation or growth of others.

• If behavior can damage themselves (i.e., cause burnout).

Suggest starting something:

• If you felt something was missing from their end in the most recent touch points.

• If you see professional growth opportunities, you think the person should take.

Improve on giving feedback

People are the most valuable asset of most businesses. Maintaining a healthy feedback culture can grow that value further and give the people involved a sense of growth and accomplishment. Keep learning how to do it, practice giving feedback, do it regularly, and request input from others.

This post is also available on medium where you can follow the author, clap for the post or comment it.

”Redis Inventory” is a tool I developed as an open-source project to practice some coding skills and also solve a very practical problem. It collects memory usage information from Redis instance (or a cluster), detects key patterns, and displays memory usage in a hierarchical way.

Pretty much like lots of tools helping you to clean and organize your hard drive (for example “Disk Inventory X” or Unbuntu’s “Disk Usage Analyzer”) it even visualizes usage information in a similar fashion as the latest (sunburst diagram). But there are some nuances. Let’s look closer!

So what is exactly the problem it is solving?

One can argue that, unlike hard disk, cache server is not persistent storage, so why bother analyzing its usage? Yeah, in theory, the cache is completely ephemeral and any application should be able to start and work with it in a “cold” state. But in reality, under the load, it is not always possible to flush the cache with no performance regression. Also, if there is a problem with how the app uses Redis, flushing would be only temporary mitigation as after some time the same problems will accumulate again. Sometimes, you just see general key count or memory consumption increase in Redis metrics, but it is not obvious where the problem is, so is hard to fix it in the code with no prior investigation.

The two most popular issues I’ve seen with cache were: cache key leak and forgetting to set TTLs or setting them too generously. Key leak happens when you accidentally add something too dynamic to the key, for example, timestamp or its hash. With TTLs you may rely on the application to delete them but miss that in some situations it will not happen and keys will stay in cache forever. Those issues are hard to track in big applications which are changing rapidly. Analysing all the code changes that can cause it with no hint for problematic keys can take days.

How the tool works

To analyze memory usage the tool scans keys space and measures each individual key size with MEMORY USAGE command. It builds a tree of nested nodes, similar to folder structure on your disk. But how do we interpret plain string keys as a hierarchy? It is quite common to use all sorts of prefixing in cache keys, we just need to reverse it. The simplest is to use a set of “separator” characters and break the string into a tuple of string segments.

Then treating each of these segments as a “folder” build a tree of nodes:

This data structure is known as “prefix tree” or “trie” (https://en.wikipedia.org/wiki/Trie) and there is another nice article about it I’ve used as a source when working on this project (https://medium.com/amboss/prefix-trees-in-action-404a6105b6d5). This structure is very convenient in this particular case for several reasons: it is thrifty for memory, it is easy to add keys there, and it is possible already at the stage of building it to aggregate usage data on each level. We can attach a container for aggregated metrics to each node. When adding a new key to the tree we will descend the tree and add value (memory usage or another) on each level in the path.

Except, as you may see, the “prod:” segment. But that’s ok as it does not introduce a fork in the tree and values there are inherently equal to the values of the nested “user:” segment. Once a new key will come (for example “prod:blog:1”) and the fork will be introduced we can clone the aggregator from the next nested node that has it.

Memory optimizations

This data structure is already compact and functional but sometimes caches may have thousands or millions of entities with the same prefix. In this case, we usually don’t care already about each individual entity id but rather want to stop at the level where they start branching.

For this case, maxChildren parameter is implemented. If the number of children of any particular node is greater than maxChildren value it will stop creating child nodes and just aggregate values at that level, also incrementing overflowChildrenCount counter so you can at least know how many keys are “lost”.

Rendering

Currently, there are two ways to render that data: text table and sunburst chart. The table view is displayed right in the terminal and has some formatting adjustments (see docs for more info).

The sunburst chart is rendered on an HTML page served by a built-in web server. Hovering any segment displays additional information as a tooltip. It is also zoomable: clicking on a segment makes it a central piece and renders sub-nodes around it.

“Quality of life” features

Here are some nice features that are not super necessary but can make your life easier while using this tool.

Custom separators

Not every application uses the colon as a separator for key segments. Other common separators are, for example, underscores, dashes, or dots. To use custom separators you can provide them as a single string in --separators option (i.e. --separators=":_-.". Don’t worry if some of them are not always used as separators: as you have seen, nodes with just one child do not introduce major overhead. One can even implement a completely custom segmentation strategy by implementing Splitter interface, for example, detecting numeric parts of the key and capturing them in a separate segment even if they are not padded with separator characters. This would require changing the code and building your own binary though.

Throttling

Another useful option is --throttle allowing you to specify the number of nanoseconds the tool will wait between requests to the Redis server. It may be handy if you are afraid to cause denial of service on your Redis instance.

Caching index result

As the tool supports different display formats, you may try some of them before you will find the most suitable. When using inventory command the tool will re-index the database every time. It may be time-consuming and can create unnecessary additional load for the server.

To avoid re-indexing every time instead of running inventory command you can use index command once to index Redis DB and save the index as a temp file. After that you can run display command several times with various parameters, it will use cached data and will not send any queries to the Redis server anymore. Keep in mind that options like maxChildren influence the trie structure and cannot be changed on the display step.

Open-source infrastructure

Here is the high-level overview of the project I have worked on lately. Although the main goal (except for creating the tool I needed) was to practice my skills, I’ve learned something about OSS projects infrastructure additionally. I was quite excited to know that I’m able to use a production-grade toolchain while working on this project absolutely for free:

• GitHub - for version-controlled hosting my code and simplistic project management;

• Travis CI - for continuous integration and delivery (uploading artifacts in docker registry);

• Coveralls - to keep an eye on unit test coverage;

• DockerHub - a public docker image registry for the dockerized version of the tool;

• Go Report Card - for static code analysis (7 linting tools).

Not to mention open-source libraries used in the project:

• Radix - a full-featured Redis client for Go,

• Cobra - CLI applications framework,

• Go-pretty - set of tools to render text tables and progress bars,

• … and a couple of smaller ones.

Also special thanks to AnyChart for issuing a free open-source license for the project.

My point here, if you want to contribute to the open-source software world, you don’t have an excuse not to do so nowadays. It is free, convenient, and the only thing you have to invest is your time and talent. So go ahead and create the tool you lack and share it with the world.

So I tried to create and here what I’ve got as a result: https://github.com/keratin/helm-charts.

When creating a helm chart it might be a little confusing what to start with. You can start with preparing a set of k8s configuration files for deployment, service, ingress controller and after that generalize them to the templates. Another way is to start right away with the dummy chart created with helm create command. This is convenient as you can reuse lots of trivial code from that dummy template and customize what you need.

Currently, there are 2 versions of chart format (v1 and v2) supported by helm2 and helm3 respectively. But, to be honest, the difference is not that huge. First, the way you list dependencies is different (in a separate file or directly in the Chart.yaml). Plus the distinction between applications and libraries is a new thing supported only by the format of version 2.

When you complete the chart of its part you can validate it with helm lint or compile templates with helm template command. But of course, nothing would prove that your chart works better then trying it at the real cluster. A nice discovery for me was the fact that you can use required function when you use the variable to force chart users to specify it.

While developing the chart you can deploy it form the source directory (don’t forget to run helm dep up to install dependencies in that case) but for distribution, you need to package it and provide a public URL to download it.

The easiest option for open-source charts is GitHub Pages. I’ve created a separate repository with GitHub Pages enabled for the master branch to host my packages charts: https://github.com/obukhov/keratin-helm-repo.

To create the package you can just run helm package <path> with the path pointing to the chart source folder. The file will be automatically named accordingly with the chart name and version. Move this file to the GitHub Pages enabled repository (in the subfolder with chart name). And finally, run the following command to generate chart index file:

helm repo index <chart folder name> --url <base url>

Base url for github pages follows the template:

https://<user>.github.io/<repo name>/<chart folder name>

That’s it. Now the repository can be added and chart can be installed from it:

helm repo add keratin https://keratin.github.io/helm-repo/charts/

helm upgrade --install <release name> keratin/keratin-authn-server

The next logical step would be to add your repository to the index of hub.helm.sh as described here: https://github.com/helm/hub/blob/master/Repositories.md to make it easier to find by other people.

Through trial and error, I was able to find a solution that works perfectly for me.

To start we will need:

• Terraform client,

• Digitalocean account,

• and… that’s it.

I would also recommend using a version control system.

Installing Terraform

Terraform is the tool to provision cloud providers. It is an opensource project maintained by Hashicorp. It is really easy to start using it. It supports plenty of cloud hosting providers (AWS, GCP, DigitalOcean, etc.) and SaaS services (GitHub, Datadog, sentry, etc.).

To install it just download a single binary file for your platform and move it to a directory included in your system’s PATH.

Registering Digitalocean account

I think I don’t have to teach you how to register. You can use my referral link to get a $100 registration bonus. That would be more than enough to experiment with Kube and decide if you want to continue using it. I obviously also will benefit from that (that’s why it is called “referral”).

After completing the registration process navigate to the API side menu item and create a new personal access token. Give it a memorable name (i.e. terraform) and grant with read/write permissions. Copy access token value after generation - you will need it at the next step.

Start coding

Now the most exciting part - create an empty directory for your project and open the editor.

Terraform provider configuration

Most files for Terraform have .tf extension. Let’s start with defining provider and access credentials for it - in this case, that would be Digitalocean. Create provider.tf file with the following content:

// provider.tf

provider "digitalocean" {
  token   = "<your access token>"
}

Is it that simple? Well, yes and no. This is the minimal provider declaration but the recommended way is to extract dynamic data like your API key to a variable, so you can change it without changing the code. Let’s make it right from the beginning and change file contents to:

// provider.tf

variable "do_token" {}

provider "digitalocean" {
  token   = var.do_token
}

The default values for the variables you can specify in a file named terraform.tfvars. And in this case in unlike the previous one the file name matters, so keep it.

// terraform.tfvars

do_token = "<your access token>"

Describing Kubernetes resources

That’s already enough to make terraform access Digitalocean API to create and destroy resources on your behalf. So let’s describe the first one to deploy the Kubernetes cluster.

// kube-cluster.tf

resource "digitalocean_kubernetes_cluster" "my_cluster" {
  name   = "my-cluster"
  region = "fra1"

  version = "1.16.2-do.3"

  node_pool {
    name       = "worker-pool"
    size       = "s-1vcpu-2gb"
    node_count = 2
  }
}

In this resource definition, you provide all necessary information for terraform to create the cluster via Digitalocean API. It is important to understand the difference between my_cluster and my-cluster: the first name exists only in terraform codebase, you can use it to refer to the attributes of the resource. The second one is the name of a cluster given in Digitalocean, you will also see it in the administrative panel. They, of course, don’t have to look alike but in most cases it makes sense.

Region parameter defines the physical location of underlying infrastructure and version is the version of Kubernetes being deployed. The following parameters define type and quantity of nodes in the cluster.

You can get the list of available options for node sizes and Kubernetes versions by installing doctl command-line tool and executing the following commands:

$ doctl kubernetes options versions

Slug           Kubernetes Version
1.16.2-do.3    1.16.2
1.15.5-do.3    1.15.5
1.14.8-do.3    1.14.8


$ doctl compute size list                                                                                  ]

Slug               Memory    VCPUs    Disk    Price Monthly    Price Hourly
512mb              512       1        20      5.00             0.007440
s-1vcpu-1gb        1024      1        25      5.00             0.007440
1gb                1024      1        30      10.00            0.014880
s-1vcpu-2gb        2048      1        50      10.00            0.014880
s-1vcpu-3gb        3072      1        60      15.00            0.022320
s-2vcpu-2gb        2048      2        60      15.00            0.022320
s-3vcpu-1gb        1024      3        60      15.00            0.022320
2gb                2048      2        40      20.00            0.029760
s-2vcpu-4gb        4096      2        80      20.00            0.029760
...

Be aware that not all droplet sizes are possible to use as Kubernetes cluster node. In fact, s-1vcpu-2gb is the smallest and cheapest one.

The number of nodes is for your consideration: a minimum of 2 nodes is required to prevent downtime during upgrades or maintenance. If you don’t mind having downtimes (as it is not a production system) you can go with one. But having one node will not provide you real experience with containers and requests being balanced to different nodes. I recommend running at least 2 nodes, which I’ve used to estimate price.

Applying changes and validating the result

The short coding part is now over and we can proceed to the testing process. First, you have to initiate terraform project once to download the provider plugin. To do that execute terraform init in the project folder.

If there are no errors run terraform apply command and confirm changes by typing yes when prompted. It can take several minutes to complete but you will be provided with the feedback:

digitalocean_kubernetes_cluster.my_cluster: Creating...
digitalocean_kubernetes_cluster.my_cluster: Still creating... [10s elapsed]
digitalocean_kubernetes_cluster.my_cluster: Still creating... [20s elapsed]
...
digitalocean_kubernetes_cluster.my_cluster: Still creating... [6m40s elapsed]
digitalocean_kubernetes_cluster.my_cluster: Creation complete after 6m43s [id=1608898b-fdcf-411a-b8e2-0e2a7820539c]

The cluster is created and we can see it in the administrative panel.

Digitalocean deploys some software to the cluster so you will not need to deploy it yourself. Kubernetes Dashboard is one of them. With the help of it, you can see deployed payloads, secrets, services, ingresses, etc. To access it, click on the link “Kubernetes Dashboard” in the cluster description page.

Terraform state file

After terraform started to communicate to Digitalocean API the new file is created in the folder terraform.tfstate, this file tracks state of the changes applied basically connecting your resource definitions to ids issued by Digitalocean for the created resources. These ids are used to update and delete resources.

To delete created cluster execute terraform destroy and confirm when prompted. Congrats! Now you can create and delete cluster with one command whenever you want to experiment with it.

Final touches

If you want to deliver an additional 10% for your codebase you can make sure it is formatted properly. Execute terraform fmt. It will have no output if all your files formatted correctly and output modified file names otherwise.

To guarantee your declaration to work in the future you can add explicit provider version definition, this will change the provider definition to following:

provider "digitalocean" {
  version = "~> 1.12"
  token   = var.do_token
}

Make sure you haven’t broken any formatting with this by running fmt again.

Conclusion

These 18 lines of code are a small step for the developer, but it is a big step for your infrastructure marking the first steps to the infrastructure-as-code approach and making it extremely easy and fast to deploy it and clean it up with no UI interactions.

Let’s take a look at how Golang can help in a simple and practical case involving copying large amounts of Redis keys.

At some point it became necessary to split our Amazon ElastiCache store into two parts — one for storing cached data, and the other for storing users’ sessions.

We, unfortunately, had them on the same instance previously. We also didn’t want to interrupt long-living sessions by resetting the storage.

Amazon ElastiCache is compatible with the Redis protocol, though with certain limitations. Redis supports the MIGRATE command, allowing you to move keys matched by a pattern from one instance to another.

Internally it works by executing DUMP+DEL commands on the source instance and creating them in target instance using RESTORE. However, Amazon’s version didn’t support this command at the time.

Back then, my practical experience with Golang was limited. I’d only implemented projects for fun and was familiar with basic syntax and concepts like goroutine and channels. But I’d decided that was enough to make use of Golang’s strengths to solve the problem I was facing.

Step 1: Let’s Write Some Simple Code

Let’s assume that Golang is fast enough to do the job. Keep in mind that Redis is, mostly, a single-threaded server from the point of view of commands execution and implements replication with no concurrency.

Preparation

I’ve picked two base libraries for this challenge:

• Radix to connect to the Redis API, and

• Cobra to make it easier to build the command-line interface for the tool

package cmd

import (
  "github.com/mediocregopher/radix/v3"
  "github.com/spf13/cobra"
  "log"
)

var pattern string
var scanCount, report, limit int

var copyCmd = &cobra.Command{
  Use:   "copy [sourceHost:port] [targetHost:port]",
  Short: "Copy keys from source redis instance to destination by given pattern",
  Long: "",
  Args:  cobra.MinimumNArgs(2),
  Run: func(cmd *cobra.Command, args []string) {
    clientSource, err := radix.DefaultClientFunc("tcp", args[0])
    if err != nil {
      log.Fatal(err)
    }

    clientTarget, err := radix.DefaultClientFunc("tcp", args[1])
    if err != nil {
      log.Fatal(err)
    }

    // ... here the copying will happen
  },
}

var rootCmd = &cobra.Command{
  Use:   "go-redis-migrate",
  Short: "Application to migrate redis data from one instance to another",
  Long:  "",
}

func init() {
  rootCmd.AddCommand(copyCmd)

  copyCmd.Flags().StringVar(&pattern, "pattern", "*", "Match pattern for keys")
  copyCmd.Flags().IntVar(&scanCount, "scanCount", 100, "COUNT parameter for redis SCAN command")
  copyCmd.Flags().IntVar(&report, "report", 1000, "After what number of keys copied to report time")
  copyCmd.Flags().IntVar(&limit, "limit", 0, "After what number of keys copied to stop (0 - unlimited)")
}

The interface is ready, it supports the “pattern” parameter to match keys, and the “limit” parameter to define the maximum number of keys. The source and destination are provided as arguments and are also required.

The Main Loop

Radix supports creating a “scanner” structure that helps you iterate over keys:

scanOpts := radix.ScanOpts{
  Command: "SCAN",
  Count:   scanCount,
}

if pattern != "*" {
  scanOpts.Pattern = pattern
}

scanner := radix.NewScanner(clientSource, scanOpts)

var key string
counter := 0

for scanner.Next(&key) {
  // copy the key

  counter++
}

if err := scanner.Close(); err != nil {
  log.Fatal(err)
}

log.Printf("In total %d keys copied in %s", counter, time.Since(start))

The loop is now ready. What’s left is to read and restore data in the target. I joined the PTTL and DUMP command to fetch time to live and value of the key in a pipeline to save execution time.

var value string
var ttl int

p := radix.Pipeline(
  radix.Cmd(&ttl, "PTTL", key),
  radix.Cmd(&value, "DUMP", key),
)

if err := clientSource.Do(p); err != nil {
  panic(err)
}

if ttl < 0 {
  ttl = 0
}

err = clientTarget.Do(radix.FlatCmd(nil, "RESTORE", key, ttl, value, "REPLACE"))
if err != nil {
  log.Fatal(err)
}

That’s already enough for the code to work, but adding some reporting logic would definitely improve the user experience.

The complete code can be found here: https://github.com/obukhov/go-redis-migrate/blob/v1.0/cmd/copy.go

But is it really that good?

Let’s run some benchmark tests by quickly spawning two Redis instances locally with Docker, and seeding the source with data (453,967 keys in total, but we only copy part of them by matching a pattern).

Then, we run each test three times to see the random deviation:

10000 keys to copy: 17.79s 18.01s 17.98s
367610 keys to copy: 8m57.98s 8m44.98s 8m58.07s

That’s not bad, but let’s see if we can improve it.

Step 2. Utilize Concurrency

Let’s visualize the sequence of operations in the current implementation:

What can we do to improve the performance here?

We can clearly see the following shortcomings:

• Reading from the source and writing to the target is serialized, although it can be executed in parallel

• The single-threaded nature of Redis only affects command execution, but serving data (network i/o) can also be parallelized. Depending on the value size, this can make a big difference

Sending multiple concurrent requests could be a good strategy. They will block each other on processing but will better utilize the i/o. This applies to both dumping and restoring data.

There is one process that can’t be parallelized— scanning the database. It relies on the scanning cursor, and there’s no way to perform the scan in multiple threads.

Let’s split the process into three stages:

• Scanning,

• Dumping data, and

• Restoring data

Scanned keys can be served through a channel to a set of goroutines, concurrently dumping these keys values and TTLs, and sending them through another channel to another set of goroutines. They, in turn, restore this data in the target instance.

Here’s an example of a visualization:

The Go gopher in this image was created by Renee French, the image was composited by me.

Gophers running in circles are loops in the goroutines, reading from the channel, processing data, and sending the processed data to another channel.

Implementation

We’ll start implementing a scanner and exporter in the same package for simplicity, starting with declaring structures:

type KeyDump struct {
  Key   string
  Value string
  Ttl   int
}

type RedisScannerOpts struct {
  Pattern          string
  ScanCount        int
  PullRoutineCount int
}

type RedisScanner struct {
  client      radix.Client
  options     RedisScannerOpts
  keyChannel  chan string
  dumpChannel chan KeyDump
}

func NewScanner(client radix.Client, options RedisScannerOpts) *RedisScanner {
  return &RedisScanner{
    client:      client,
    options:     options,
    dumpChannel: make(chan KeyDump),
    keyChannel:  make(chan string),
  }
}

Two channels are declared here. The first is a plain string channel to send scanned keys from the scanner to the group of exporting goroutines. The second channel of KeyDumpstructures is for sending dumped data to the goroutines restoring data.

A KeyDump structure contains all the necessary information about simple Redis values: key, value, and TTL.

First Goroutines

The following function orchestrates goroutines for scanning and exporting data:

func (s *redisScanner) Start() {
  wgPull := new(sync.WaitGroup)
  wgPull.Add(s.options.PullRoutineCount)

  go s.scanRoutine()
  for i := 0; i < s.options.PullRoutineCount; i++ {
    go s.exportRoutine(wgPull)
  }

  wgPull.Wait()
  close(s.dumpChannel)
}

As you can see, it spawns one scanning routine and the number of exporting goroutines defined by the PullRoutineCount option. Pay attention to the variable named wgPull of type WaitGroup, a handy tool that makes sure our code doesn’t exit before the process is complete.

WaitGroup waits for a collection of goroutines to finish. The main goroutine calls Add to set the number of goroutines to wait for. Then each of the goroutines runs and calls Done when finished. At the same time, Wait can be used to block excution until all goroutines have finished.

Waitgroup is initialized through the Add method with the overall number of goroutines. The variable is provided to each goroutine as an argument, and when goroutine finishes its work, it calls Done method. Add increments an internal counter andDone decrements it. Wait method blocks execution until the counter reaches zero.

The goroutine scanner structure is similar to what we had in the first version:

func (s *RedisScanner) scanRoutine() {
  var key string
  scanOpts := radix.ScanOpts{
    Command: "SCAN",
    Count:   s.options.ScanCount,
  }

  if s.options.Pattern != "*" {
    scanOpts.Pattern = s.options.Pattern
  }

  radixScanner := radix.NewScanner(s.client, scanOpts)
  for radixScanner.Next(&key) {
    s.keyChannel <- key
  }

  close(s.keyChannel)
}

Everything is self-explanatory, but there are a few things worth mentioning:

• <- sends data to the channel

• At the very end, we close the channel to let the goroutines know that no more data is going to be sent

Exporting goroutine already looks familiar to us. Terminating execution on a client error is not really a nice way to handle errors, but if the connection is reliable, it will never be a problem.

func (s *RedisScanner) exportRoutine(wg *sync.WaitGroup) {
  for key := range s.keyChannel {
    var value string
    var ttl int

    p := radix.Pipeline(
      radix.Cmd(&ttl, "PTTL", key),
      radix.Cmd(&value, "DUMP", key),
    )

    if err := s.client.Do(p); err != nil {
      log.Fatal(err)
    }

    if ttl < 0 {
      ttl = 0
    }

    s.reporter.AddExportedCounter(1)
    s.dumpChannel <- KeyDump{
      Key:   key,
      Ttl:   ttl,
      Value: value,
    }
  }

  wg.Done()
}

The reading from the channel is implemented with the range keyword which exits the for loop automatically when the channel (s.keyChannel) is closed. wg.Done() in the last line helps to ensure that all the keys passed through s.keyChannel were dumped and sent through s.dumpChannel.

As you maybe know, struct fields starting with a lower-case letter are considered internal for the package, so we have to provide a getter in order to allow other packages to read the dumpChannel field. This is also a chance to declare return type as a channel only intended to be read from (using <-chan instead of just chan type):

func (s *RedisScanner) GetDumpChannel() <-chan KeyDump {
  return s.dumpChannel
}

Goroutines to Restore Exported Data

Pusher can also be configured and uses WaitGroup to orchestrate goroutines:

func NewRedisPusher(client radix.Client, dumpChannel <-chan scanner.KeyDump) *RedisPusher {
  return &RedisPusher{
    client:      client,
    dumpChannel: dumpChannel,
  }
}

type RedisPusher struct {
  client      radix.Client
  dumpChannel <-chan scanner.KeyDump
}

func (p *RedisPusher) Start(wg *sync.WaitGroup, number int) {
  wg.Add(number)
  for i := 0; i < number; i++ {
    go p.pushRoutine(wg)
  }
}

And pushRoutine uses a similar practice to read from the channel and exit:

func (p *RedisPusher) pushRoutine(wg *sync.WaitGroup) {
    for dump := range p.dumpChannel {
        p.reporter.AddPushedCounter(1)
        err := p.client.Do(radix.FlatCmd(nil, "RESTORE", dump.Key, dump.Ttl, dump.Value, "REPLACE"))
        if err != nil {
        log.Fatal(err)
        }
    }

    wg.Done()
}

There is one important thing to note here: dumpChannel is closed by scanner only after all exporters exit. That guarantees no data will be lost at the very end. It’s achieved with wgPull and two lines in RedisScanner‘s Start() receiver:

wgPull.Wait()
close(s.dumpChannel)

Wiring Everything Together

Now let’s use developed packages in the Cobra command to put it all together.

First, extend the command definition to add more options:

func init() {
  rootCmd.AddCommand(copyCmd)
  copyCmd.Flags().StringVar(&pattern, "pattern", "*", "Match pattern for keys")
  copyCmd.Flags().IntVar(&scanCount, "scanCount", 100, "COUNT parameter for redis SCAN command")
  copyCmd.Flags().IntVar(&report, "report", 1, "Report current status every N seconds")
  copyCmd.Flags().IntVar(&exportRoutines, "exportRoutines", 30, "Number of parallel export goroutines")
  copyCmd.Flags().IntVar(&pushRoutines, "pushRoutines", 30, "Number of parallel push goroutines")
}

Then, create a scanner and pusher (and WaitGroup for them). Don’t forget to call Wait() on it, otherwise, the command will exit immediately:

// clientSource and clientTarget initialization
redisScanner := scanner.NewScanner(
  clientSource,
  scanner.RedisScannerOpts{
        Pattern:          pattern,
        ScanCount:        scanCount,
        PullRoutineCount: exportRoutines,
    },
)

redisPusher := pusher.NewRedisPusher(clientTarget, redisScanner.GetDumpChannel())

waitingGroup := new(sync.WaitGroup)

redisPusher.Start(waitingGroup, pushRoutines)
redisScanner.Start()

waitingGroup.Wait()

Benchmark

The most exciting part is to see the difference. Let’s take a look at the same test cases and compare them:

Test #1

Source database: 453,967 keys.Keys to copy: 10,000 keys.

Test #2

Source database: 453,967 keys.Keys to copy: 367,610 keys.

Processing is three to nine times faster when testing on a local machine. The real execution on the production infrastructure took less than two minutes to copy about a million keys.

Conclusion

This application was one of my first codebases written in Golang. It also helped to fix a real-life problem with very little development and operations time needed.

If you look at the full version of the code here, you’ll see a “sidecar” goroutine that collects counters on scanned, exported, and pushed keys, and reports in with configured time intervals to stdout. It helps to see the progress of execution in the following format:

Start copying
2021/02/14 13:11:42 Scanned: 29616 Exported: 29616 Pushed: 29616 after 1.000153648s
2021/02/14 13:11:43 Scanned: 59621 Exported: 59615 Pushed: 59615 after 2.000128223s
2021/02/14 13:11:44 Scanned: 89765 Exported: 89765 Pushed: 89765 after 3.0001194s
2021/02/14 13:11:44 Scanned: 100000 Exported: 100000 Pushed: 100000 after 3.347127281s
Finish copying

Do you have some examples of how Golang helped you to find a simple solution to a tricky problem? Tweet me a response with the link below.

Our team has been working for the past 18 months on porting AMBOSS functionality to GraphQL API. During this time we faced some technical and organizational challenges. The talk describes the experience of this migration process, what caused questions, and how we as a team approached them.

The slides can be downloaded here

The talk and slides are in English. There were five speakers in total: two from Berlin and three from Wroclaw. More details are on the event’s page at meetup.com.

Recording

Photos