Practical use of artificial intelligence in software architecture

Advertisement

Pandora’s box is open: Generative AI in the form of large language models (LLMs) such as ChatGPT, Gemini, Cloudy, etc. has developed rapidly and is conquering more and more areas in the work of software developers. It can also support the activities of software architects, as some examples below will show.

One of the most important properties of today’s LLMs is that the size of the underlying neural network, measured in parameters (weights of neurons), also determines the performance. For flexible support of architectural work, a large model is beneficial, which often cannot be operated locally, which is why cloud services are indispensable. This raises questions about trust in data processing in the cloud, each provider has its own guidelines. However, the article shows that LLMs provide benefits even without revealing internal data.

7. Product Owner Day: Workshops on OKRs and High Impact Teams

Obverse and context

There are various components behind a technology like ChatGPT: LLM is just one aspect, as the front end that controls LLM is also very important. Take a look at the API (Documentation And Reference) shows the stateless character of all requests to the LLM. With each request it receives and returns a request answer. To build a dialog, the front end builds a context from questions and answers, which grows over time and is repeatedly passed to the model.

Users can take advantage of some of the features of the front end. For example, Figure 1 shows a part of the smart controls found under each part of the dialog in the ChatGPT frontend. Your own prompts can be edited here at any time (edit icon) to more precisely specify the requirement. The bot’s response can be regenerated at any time (reload icon) to get better results. This means that users actually replace the linear chat process with a dialog tree that they navigate using the <> icon (see Figure 2).

Although the size of context that new models can handle is constantly growing, many factors play an important role in the quality of answers. Just as people work with only limited short-term memory when speaking, the same is true with LLM. If users overload the model with information in context, the answers will eventually fall out of this context and the model suddenly does not know what the actual task of the dialogue was. A dialogue tree reduces context because only one branch of the tree is used for it, not the entire dialogue with all the corrections.

Rethinking an answer generally only works if the model allows changes to the answer. This is done via parameters temperature Controlled in the API. If it is greater than zero, the model becomes more creative by not always generating the most likely answer. For architectural work, it may be sensible to vary this parameter depending on the task.

The cost of the API also plays a role in this context: requests are billed in input and output tokens. One token corresponds to approximately one to four characters, so requests become more expensive with every step in the dialogue as the context grows. These costs add up very quickly. A well-formed, shorter context created by editing a prompt rather than rewriting it can yield immediate benefits. This is especially noticeable when the model generates a long response to a query. If the user rewrites the prompt rather than editing it, the long answer ends up in the context many times and generates costs.

Users should also watch out for session poisoning. If it went in the wrong direction because users gave the wrong terms or the model made a mistake in the answer, this misinformation persists in a linear chat process. In human dialogue, conversation partners would ignore it – but not a machine. The dialogue is filled with incorrect information, which later affects the quality of answers. This is why navigation within the dialogue is so important, because as soon as the user recognizes an error, he goes back to the dialogue tree and closes the branches before the error.