TimbedPå dansk

Is your software an AI system? A staircase in six steps

AI has arrived in finance, HR and CRM systems with an update. A six-step staircase shows when software becomes an AI system under the EU AI Act, what supplier and customer should be able to document, and why it is the use that decides the risk.

Managers and suppliers often put the same question to me: “We have AI in the product now. Does that bring us under the AI Act?”

The answer turns on two things that tend to get mixed up. The first is whether it is an AI system at all. The second is what it is used for. The first decides whether the regulation applies. The second decides how much it asks of you.

For the first, I use a staircase with six steps. It is not legally precise to the last comma, and it is not meant to be. It is there to help you place your own systems and ask your suppliers the right questions.

Does the system work things out, or follow rules?

The regulation calls something an AI system when it infers, from the input it receives, how to produce its output: a prediction, a piece of text, a recommendation or a decision. It may also learn while in use, but that is not a requirement.

What falls outside is software where people have written every rule. In its guidelines the Commission also leaves out basic data processing such as sorting, filtering and statistics, fixed rules of thumb of the kind a classic chess program uses, and simple predictions such as last week’s average. Its own examples are a database that finds every customer who bought a given product last month, an ordinary spreadsheet without AI features, and a sales dashboard that shows the figures without recommending what to do about them.

Nor is an AI model an AI system on its own. It becomes one when somebody builds something around it, a user interface for instance. A company that builds a product on a model from OpenAI, Anthropic or Google is therefore itself the provider of an AI system. The model provider has duties of its own, but they do not cover the product.

The staircase

Step What it is Known examples AI system?
0 Traditional software People wrote every rule. The same input gives the same output A spreadsheet without AI features, a customer database No
1 Software with automation Rules, filters, sorting, statistics, averages Rules in Outlook, fixed matching rules in bank reconciliation in Dinero and Business Central No
2 Software with one AI feature One bounded call to an AI model inside an otherwise ordinary product Gemini summaries in Gmail, Copilot in Business Central’s bank reconciliation The feature yes, the rest no
3 AI-driven application At its core is the model’s answer to the user, with instructions, data and context around it ChatGPT, Microsoft’s Copilot for organisations Yes
4 Agent Takes several steps on its own, uses tools and acts on the user’s behalf Payables Agent in Business Central, HubSpot’s Customer Agent Yes
5 Own trained model The supplier’s own model, trained on data and updated as it goes, with output that drives decisions Stripe Radar against payment fraud, Netflix recommendations Yes

The steps differ on four points: who wrote the logic, how central AI is to the product, how independently the system acts, and whether the model is finished or keeps learning.

Place the feature, not the product. The filters you set up yourself in Gmail are step 1. The spam filter uses machine learning. The Gemini summary is step 2. One product, three places on the staircase.

AI is already in the systems you use

Most companies never decided to buy an AI system. It arrived with an update. Here are some of the systems widely used in Denmark, placed according to the suppliers’ own documentation.

Finance. Microsoft’s Business Central shows the whole staircase in one system. Bank reconciliation has fixed matching rules, which Microsoft itself calls the traditional route without AI. That is step 1. Whatever the rules cannot match, Copilot can try to find, such as one payment that covers several invoices. That is step 2, and nothing is saved until the user has approved it. Payables Agent watches an inbox, reads supplier invoices and drafts purchase invoices under its own user identity. That is step 4, although the agent does not post anything itself.

Dinero, a Danish accounting system, suggests a match in bank reconciliation when the date and amount agree. That is a rule. Dinero also has an AI agent which, according to its terms, is built on language models and gives suggestions for guidance only. It is called an agent, but as far as the terms describe it, it does not act on its own, so it sits on step 2 or 3. The name tells you nothing about the step.

In Pleo, the administrator writes the expense policy in plain language and AI assesses expenses against it. People wrote the rules, but a model interprets them. In my view that makes it an AI system.

HR and recruitment. In Planday, whoever draws up the rota can choose between rule-based matching and recommendations that, according to Planday, learn from the company’s scheduling patterns. The first is step 1. Whether the second is AI in the regulation’s sense depends on how it learns, and that is a question for the supplier. Workday’s Peakon summarises employees’ comments with generative AI, a typical step 2 feature.

Recruitment systems are where it gets interesting. Teamtailor’s Co-pilot assesses whether a candidate meets the criteria for the job and can sort those who meet the most to the top. Teamtailor itself states that certain of its AI features are classified as high-risk. Talentech, which also took over what used to be HR Manager’s recruitment tool, scores applications from 1 to 10, and HR-ON describes an AI score from 1 to 5 stars.

Technically, features like these often sit on step 2: one feature that sends the application and the job ad to a language model. But the purpose is to assess applicants, and that is one of the regulation’s high-risk areas. If you use one, ask your supplier how the feature is classified and ask for the reasoning in writing. Features that only write the job ad are a different matter. They produce text and assess nobody.

CRM and customer service. SuperOffice can suggest which category a customer enquiry belongs to. The model is trained on a selection of the company’s own enquiries, and staff approve each suggestion. HubSpot’s Customer Agent answers customers on chat and other channels, carries out actions you have set up, such as booking a meeting, and hands the conversation to a person when it is unsure. That is step 4, and because it talks directly to your customers, they need to know it is an AI.

What supplier and customer should be able to document

The regulation uses two words worth knowing. The provider develops the system and places it on the market, usually the supplier. The deployer is the company that uses it in its work, in other words the customer. The third party is the provider of the general-purpose model, such as OpenAI, Anthropic or Google.

Steps 0 and 1. The AI Act asks nothing. Even so, the supplier should have a short written explanation of why the product is not an AI system, because customers ask. The customer should have placed the system, so that it does not turn up as a gap when someone asks. And bear in mind that staff are probably using ChatGPT or something like it on the side. Then you are a deployer anyway.

Step 2. The supplier should be able to show:

  • Which features use AI, and which model from which provider sits behind them
  • The agreement with the model provider: where data is processed, whether it is used for training, and how long it is kept
  • That the model provider is listed as a sub-processor in the data processing agreement, and how any transfer to the United States is covered (the Data Privacy Framework or standard contractual clauses)
  • That staff have the AI literacy needed to build and run the feature
  • Whether generated content has to be marked in a machine-readable way as artificially generated. Features that only assist with editing are exempt, so this needs a case-by-case assessment

The customer should be able to show that the people using the feature know what it can and cannot do, that it is not used for a high-risk purpose, and that the list of sub-processors is up to date. Keep an eye out when an update brings a new AI feature, because that is often how they arrive.

Step 3. The supplier should be able to show everything from step 2, and also:

  • That users are told they are dealing with an AI, unless that is obvious
  • What the system is intended for, and what it must not be used for. This is where the risk class begins
  • The documentation on the model’s capabilities and limitations that the model provider is obliged to make available
  • If the system can be used in a high-risk area: a written assessment of why it is not high-risk. A supplier that reaches that conclusion must also register the system in the EU database

The customer needs a usage policy covering what staff may use the system for and what data they may put into it. The customer should also be able to show what the system is actually used for, and why that is not high-risk. AI-generated images, video and sound that look real must be labelled, and text published to inform the public on matters of public interest must either be labelled or have been through editorial review. Where the use of personal data is likely to pose a high risk to the people concerned, data protection law requires an impact assessment.

Step 4. The regulation has no particular rules for agents. But an agent does things: it sends, saves, orders and changes. A mistake is no longer a poor answer but an action. The supplier should therefore also be able to show which tools and systems the agent can use, what it may do without asking, how its actions are logged, and how a person stops it. The customer should be able to show that the agent can reach only what it needs, who approves what it must not do alone, and that somebody reads the log.

Step 5. The supplier should also be able to show where the training data comes from and on what basis it is used, and how the model is tested and monitored as it changes. If the supplier uses customers’ data to train its own model, it is no longer acting only as the customer’s processor on that point, and the agreement needs to say so. The customer should know whether its own data is used for training, and who can overrule the system’s decisions.

The step decides whether it is an AI system. The use decides the risk class

That is the whole point of the staircase. The regulation sorts by use, not by technology:

  • Prohibited, for example reading employees’ emotions from their face or voice, or social scoring
  • High-risk, when the system is used in the areas the regulation lists: among them recruitment and people management, admission to and assessment in education, creditworthiness, insurance, access to public benefits and critical infrastructure, and AI as a safety component in products such as machinery and medical devices
  • Transparency, for chatbots and generated content
  • Everything else, where the requirements are AI literacy and ordinary good sense

Take ChatGPT. If marketing uses it to draft the newsletter, there is little to document. If HR uses it to sort applicants, the same tool is suddenly in a high-risk area. And because you changed its purpose yourselves, you may end up being treated as the provider of a high-risk system, with the high-risk requirements falling on you. It is an easy trap to walk into, and it does not lie in the technology. It lies in a decision nobody wrote down.

It works the other way too. Stripe Radar is on step 5: a model Stripe has trained on data from across its network, which blocks payments it judges to be fraudulent. Even so, it is not high-risk, because fraud detection is expressly kept out of the high-risk rules for credit scoring. The highest step does not mean the highest risk.

What applies when

The regulation was amended in July 2026, and several deadlines have moved.

From What
2 February 2025 The prohibitions and the AI literacy requirement. Since the amendment, companies must support the development of their staff’s skills, but need not guarantee any particular level
2 August 2025 The duties of providers of general-purpose models
2 August 2026 Transparency: chatbots, marking of generated content, deepfakes
2 December 2026 Marking of generated content for systems already on the market before 2 August 2026, and new bans on sexualised deepfakes made without consent
2 December 2027 The high-risk requirements for the listed areas, such as recruitment, education and credit
2 August 2028 The high-risk requirements for AI in regulated products

The Commission’s guidelines on when something is high-risk are still only a draft.

Two things you can do now

  1. Place every AI feature on the staircase, both your own and those that came with your suppliers’ updates, and write down why. It is not a big project, and it gives you the answer the next time a customer, an auditor or a board asks.
  2. Write down what you use them for, and what they must not be used for. It is the use that decides the risk class, and that is the part you control.

Sources