Skip to content

Commit

Permalink
minor ai improvements
Browse files Browse the repository at this point in the history
  • Loading branch information
DavertMik committed Aug 27, 2024
1 parent eedf001 commit e3ffa9b
Show file tree
Hide file tree
Showing 4 changed files with 44 additions and 13 deletions.
26 changes: 14 additions & 12 deletions docs/ai.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
permalink: /ai
title: Testing with AI 🪄
title: Testing with AI 🪄
---

# 🪄 Testing with AI
Expand Down Expand Up @@ -37,7 +37,7 @@ AI providers have limits on input tokens but HTML pages can be huge. However, so

Even though, the HTML is still quite big and may exceed the token limit. So we recommend using models with at least 16K input tokens, (approx. 50K of HTML text), which should be enough for most web pages. It is possible to strictly limit the size of HTML to not exceed tokens limit.

> ❗AI features require sending HTML contents to AI provider. Choosing one may depend on the descurity policy of your company. Ask your security department which AI providers you can use.
> ❗AI features require sending HTML contents to AI provider. Choosing one may depend on the descurity policy of your company. Ask your security department which AI providers you can use.


Expand Down Expand Up @@ -91,7 +91,7 @@ ai: {
model: 'gpt-3.5-turbo-0125',
messages,
});

return completion?.choices[0]?.message?.content;
}
}
Expand Down Expand Up @@ -146,7 +146,7 @@ ai: {
model: 'claude-2.1',
max_tokens: 1024,
messages
});
});
return resp.content.map((c) => c.text).join('\n\n');
}
}
Expand All @@ -167,7 +167,7 @@ ai: {
const { OpenAIClient, AzureKeyCredential } = require("@azure/openai");

const client = new OpenAIClient(
"https://<resource name>.openai.azure.com/",
"https://<resource name>.openai.azure.com/",
new AzureKeyCredential("<Azure API key>")
);
const { choices } = await client.getCompletions("<deployment ID>", messages);
Expand Down Expand Up @@ -260,7 +260,7 @@ async function makeApiRequest(endpoint, deploymentId, messages) {
}
```
### Writing Tests with AI Copilot
## Writing Tests with AI Copilot
If AI features are enabled when using [interactive pause](/basics/#debug) with `pause()` command inside tests:
Expand Down Expand Up @@ -302,11 +302,11 @@ GPT will generate code and data and CodeceptJS will try to execute its code. If
This AI copilot works best with long static forms. In the case of complex and dynamic single-page applications, it may not perform as well, as the form may not be present on HTML page yet. For instance, interacting with calendars or inputs with real-time validations (like credit cards) can not yet be performed by AI.
Please keep in mind that GPT can't react to page changes and operates with static text only. This is why it is not ready yet to write the test completely. However, if you are new to CodeceptJS and automated testing AI copilot may help you write tests more efficiently.
Please keep in mind that GPT can't react to page changes and operates with static text only. This is why it is not ready yet to write the test completely. However, if you are new to CodeceptJS and automated testing AI copilot may help you write tests more efficiently.
> 👶 Enable AI copilot for junior test automation engineers. It may help them to get started with CodeceptJS and to write good semantic locators.
### Self-Healing Tests
## Self-Healing Tests
In large test suites, the cost of maintaining tests goes exponentially. That's why any effort that can improve the stability of tests pays itself. That's why CodeceptJS has concept of [heal recipes](./heal), functions that can be executed on a test failure. Those functions can try to revive the test and continue execution. When combined with AI, heal recipe can ask AI provider how to fix the test. It will provide error message, step being executed and HTML context of a page. Based on this information AI can suggest the code to be executed to fix the failing test.
Expand Down Expand Up @@ -358,7 +358,7 @@ When execution finishes, you will receive information on token usage and code su
By evaluating this information you will be able to check how effective AI can be for your case.
### Arbitrary GPT Prompts
## Arbitrary Prompts
What if you want to take AI on the journey of test automation and ask it questions while browsing pages?
Expand Down Expand Up @@ -412,7 +412,7 @@ npx codeceptjs shell --ai
Also this is availble from `pause()` if AI helper is enabled,
Ensure that browser is started in window mode, then browse the web pages on your site.
Ensure that browser is started in window mode, then browse the web pages on your site.
On a page you want to create PageObject execute `askForPageObject()` command. The only required parameter is the name of a page:
```js
Expand All @@ -438,6 +438,8 @@ If page object has `clickForgotPassword` method you can execute it as:
=> page.clickForgotPassword()
```
Here is an example of a session:
```shell
Page object for login is saved to .../output/loginPage-1718579784751.js
Page object registered for this session as `page` variable
Expand Down Expand Up @@ -481,11 +483,11 @@ GPT prompts and HTML compression can also be configured inside `ai` section of `
```js
ai: {
// define how requests to AI are sent
// define how requests to AI are sent
request: (messages) => {
// ...
}
// redefine prompts
// redefine prompts
prompts: {
// {}
},
Expand Down
29 changes: 29 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,35 @@ layout: Section

# Releases

## 3.6.5

❤️ Thanks all to those who contributed to make this release! ❤️

🛩️ *Features*
* feat(helper): playwright > wait for disabled ([#4412](https://github.com/codeceptjs/CodeceptJS/issues/4412)) - by **[kobenguyent](https://github.com/kobenguyent)**
```
it('should wait for input text field to be disabled', () =>
I.amOnPage('/form/wait_disabled').then(() => I.waitForDisabled('#text', 1)))
it('should wait for input text field to be enabled by xpath', () =>
I.amOnPage('/form/wait_disabled').then(() => I.waitForDisabled("//*[@name = 'test']", 1)))
it('should wait for a button to be disabled', () =>
I.amOnPage('/form/wait_disabled').then(() => I.waitForDisabled('#text', 1)))
Waits for element to become disabled (by default waits for 1sec).
Element can be located by CSS or XPath.
**[param](https://github.com/param)** {CodeceptJS.LocatorOrString} locator element located by CSS|XPath|strict locator. **[param](https://github.com/param)** {number} [sec=1] (optional) time in seconds to wait, 1 by default. **[returns](https://github.com/returns)** {void} automatically synchronized promise through #recorder
```

🐛 *Bug Fixes*
* fix(AI): AI is not triggered ([#4422](https://github.com/codeceptjs/CodeceptJS/issues/4422)) - by **[kobenguyent](https://github.com/kobenguyent)**
* fix(plugin): stepByStep > report doesn't sync properly ([#4413](https://github.com/codeceptjs/CodeceptJS/issues/4413)) - by **[kobenguyent](https://github.com/kobenguyent)**
* fix: Locator > Unsupported pseudo selector 'has' ([#4448](https://github.com/codeceptjs/CodeceptJS/issues/4448)) - by **[anils92](https://github.com/anils92)**

📖 *Documentation*
* docs: setup azure open ai using bearer token ([#4434](https://github.com/codeceptjs/CodeceptJS/issues/4434)) - by **[kobenguyent](https://github.com/kobenguyent)**

## 3.6.4

❤️ Thanks all to those who contributed to make this release! ❤️
Expand Down
1 change: 0 additions & 1 deletion docs/community-helpers.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ Please **add your own** by editing this page.
* [codeceptjs-slack-reporter](https://www.npmjs.com/package/codeceptjs-slack-reporter) Get a Slack notification when one or more scenarios fail.
* [codeceptjs-browserlogs-plugin](https://github.com/pavkam/codeceptjs-browserlogs-plugin) Record the browser logs for failed tests.
* [codeceptjs-testrail](https://github.com/PeterNgTr/codeceptjs-testrail) - a plugin to integrate with [Testrail](https://www.gurock.com/testrail)
* [codeceptjs-monocart-coverage](https://github.com/cenfun/codeceptjs-monocart-coverage) - a plugin to generate coverage reports, it integrate with [monocart coverage reports](https://github.com/cenfun/monocart-coverage-reports)

## Browser request control
* [codeceptjs-resources-check](https://github.com/luarmr/codeceptjs-resources-check) Load a URL with Puppeteer and listen to the requests while the page is loading. Enabling count the number or check the sizes of the requests.
Expand Down
1 change: 1 addition & 0 deletions lib/template/heal.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ heal.addRecipe('ai', {
prepare: {
html: ({ I }) => I.grabHTMLFrom('body'),
},
suggest: true,
steps: [
'click',
'fillField',
Expand Down

0 comments on commit e3ffa9b

Please sign in to comment.