245 lines
11 KiB
Markdown
245 lines
11 KiB
Markdown
# Prerequisites
|
|
|
|
Before we get into writing tests, please make sure you have pre-commit hook for
|
|
styling tools setup so CI won't fail from these
|
|
|
|
Instructions [here](http://git.jingrow.com/jingrow/jcloude/issues/424#issuecomment-1193375098)
|
|
|
|
# Writing Tests for Press
|
|
|
|
Writing tests involve running tests locally (duh). So let's get that setup. (You'll only have to do this once)
|
|
|
|
## Make a test site
|
|
|
|
Tests can leave fake records. This will pollute your local setup. So, get
|
|
yourself a test site. You can get these commands from the CI workflow file too,
|
|
but I'll save you some time. You can name the site and set password to whatever.
|
|
|
|
```sh
|
|
bench new-site --db-root-password admin --admin-password admin test_site
|
|
bench --site test_site install-app press
|
|
bench --site test_site add-to-hosts # in case you wanna call APIs
|
|
bench --site test_site set-config allow_tests true
|
|
```
|
|
|
|
Finally, you need to start bench as some of the tests may want to trigger
|
|
background jobs, which would fail if background workers aren't there
|
|
|
|
```sh
|
|
bench start
|
|
```
|
|
|
|
As you write tests you'll occasionally want to remove all test data
|
|
in your test site from time to time. So, here ya go:
|
|
|
|
```sh
|
|
bench --site test_site reinstall --yes
|
|
```
|
|
|
|
## Writing tests
|
|
|
|
This is the hard part. Because of Press's dependency with outside world, it's
|
|
hard to isolate unit tests to this project. Regardless it's still possible with
|
|
plain old python's built in libraries.
|
|
|
|
Majority of this is done with the help of python's `unittest.mock` library. We
|
|
use this library to mock parts of code when referencing things that are out of
|
|
Press's control.
|
|
|
|
Eg: We can mock all Agent Job creation calls by decorating the TestCase class like so
|
|
|
|
```python
|
|
@patch.object(AgentJob, "enqueue_http_request", new=Mock())
|
|
class TestSite(FrappeTestCase):
|
|
```
|
|
|
|
We use `patch.object` decorator here so that every instance of `AgentJob`
|
|
object will have it's `enqueue_http_request` method be replaced by whatever we
|
|
pass in the new argument, which in this case is `Mock()` which does nothing.
|
|
You can think of it as a `pass`. But it has other uses as you'll find if you
|
|
keep reading.
|
|
|
|
> Note: Class decorators aren't inherited, so you'll have to do this on all
|
|
> classes you want to mock http request creation for Agent Job
|
|
|
|
## Mocking Agent Jobs End-to-end
|
|
|
|
There's also a decorator you can use to fake the result of an agent job. For
|
|
example, you may do it like so:
|
|
http://git.jingrow.com/jingrow/jcloude/blob/983631ccb59f88e57fd60fdad1615e9abd87d99f/press/api/tests/test_site.py#L243-L247
|
|
|
|
This way you can use the name of the type of job and fake a response from the same.
|
|
|
|
You may also fake the output obtained from the job which you can then use to test the callback that uses the same:
|
|
http://git.jingrow.com/jingrow/jcloude/blob/983631ccb59f88e57fd60fdad1615e9abd87d99f/press/api/tests/test_site.py#L305-L323
|
|
|
|
It is also possible to fake multiple jobs in the same context, for when multiple jobs are processed in the same request or job:
|
|
|
|
http://git.jingrow.com/jingrow/jcloude/blob/983631ccb59f88e57fd60fdad1615e9abd87d99f/press/press/doctype/site_migration/test_site_migration.py#L29-L77
|
|
|
|
> Note that with this, you can't fake 2 results for the same type of job. This is still a limitation. As a workaround, you can have multiple `with` statements for such cases.
|
|
|
|
This is all done with the help of the [responses](http://git.jingrow.com/getsentry/responses) library by intercepting the http requests for the same.
|
|
|
|
> Note that you shouldn't mock `AgentJob.enqueue_http_request` when using the above decorator as that will interfere with the request interception need to fake the job results
|
|
|
|
Now that we've learned to mock the external things, we can go about mocking
|
|
internal things, which forms the basis of testing, which is
|
|
|
|
1. Make test records
|
|
2. Perform operation (i.e Run code that will on production)
|
|
3. Test the test records for results
|
|
|
|
### Making test records
|
|
|
|
Making test records is also kind of a pain as we have validations all around
|
|
code that will need to be passed every time you create a pg. This is too much
|
|
cognition. Therefore, we can create utility functions (with sensible defaults)
|
|
to make test record of the corresponding Doctype in their own corresponding
|
|
test files (for organization reasons). These functions will be doing the bare
|
|
minimum to make a valid document of that doctype.
|
|
|
|
Eg: `create_test_bench` in `test_bench.py` can be imported and used whenever
|
|
you need a valid bench (which itself has dependencies on many other doctypes)
|
|
|
|
You can also add default args to these utility functions as you come across the
|
|
need. Just append to end so you won't have to rewrite pre-existing tests.
|
|
|
|
You write a test by writing a method in the TestCase. Make the method name as
|
|
long as you want. Test methods are supposed to test a specific case. When the
|
|
test breaks eventually (serving it's purpose), the reader should be able to
|
|
tell what it's trying to test is supposed without even having to read the code.
|
|
Making the method name small is pointless; we're never going to reference this
|
|
method anywhere in code, ever. Eg:
|
|
|
|
http://git.jingrow.com/jingrow/jcloude/blob/2503e523284fb905eca60acf3271d3fb1dccbc3f/press/press/doctype/site/test_site.py#L215-L228
|
|
|
|
You can also go the extra mile and write a function docstring. This docstring
|
|
will be shown in the output when the testrunner detects that the test has
|
|
failed.
|
|
|
|
### Rerunnability
|
|
|
|
Not a real word, but I like to be able to re-run my tests without having to
|
|
nuke the database. Leaving the database in an "empty state" after every test is
|
|
a very easy way to achieve this. This also makes testing for things like count
|
|
of docs super easy. Lucky for us there's a method in `TestCase` that's run
|
|
after every individual test in the class. It's called `tearDown`.
|
|
|
|
We can easily do
|
|
|
|
```python
|
|
def tearDown(self):
|
|
frappe.db.rollback()
|
|
```
|
|
|
|
And every pg you create (in foreground at least) will not be committed into the database.
|
|
|
|
> Note: If the code you're testing calls frappe.db.commit, be sure to mock it
|
|
> cuz otherwise docs will get committed till that point regardless.
|
|
|
|
You can mock certain lines while testing a piece of code with the `patch` decorator too. Eg:
|
|
|
|
```python
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
# this will mock all the frappe.db.commit calls in server.py while in this test suite
|
|
@patch("press.press.doctype.server.server.frappe.db.commit", new=MagicMock)
|
|
class TestBench(FrappeTestCase):
|
|
```
|
|
|
|
You can also use the patch decorator on test methods too. Eg:
|
|
|
|
http://git.jingrow.com/jingrow/jcloude/blob/6dd6b2c8193b04f1aec1601d52ba09ce9dca8dfe/press/tests/test_cleanup.py#L280-L290
|
|
The decorator passes the mocked function (which is a `Mock()` object) along as
|
|
an argument, so you can later do asserts on it (if you want to).
|
|
|
|
You can even use the decorator as context manager if you don't want to mock
|
|
things for the entirety of the test.
|
|
|
|
http://git.jingrow.com/jingrow/jcloude/blob/6dd6b2c8193b04f1aec1601d52ba09ce9dca8dfe/press/tests/test_audit.py#L97-L102
|
|
|
|
Here, we're actually faking the output of the function which usually calls a
|
|
remote endpoint that's out of our control by adding the `new` argument to the
|
|
method.
|
|
|
|
> Note: When you use asserts on Mock object, Document comparisons will mostly
|
|
> work as expected as we're overriding \***\*eq\*\*** of Document class during
|
|
> tests (check before_test.py). This is because by default when 2 Document
|
|
> objects are compared, only their `id()` is checked, which will return False
|
|
> as the objects will be different in memory.
|
|
|
|
> Note: If you need to mock some Callable while preserving it's function, (in
|
|
> case you want to do asserts on it, you can use the `wraps` kwarg instead of
|
|
> new). Eg:
|
|
|
|
http://git.jingrow.com/jingrow/jcloude/blob/23711e2799f2d24dfd7bbe2b6cd148f54f4b253b/press/press/doctype/database_server_mariadb_variable/test_database_server_mariadb_variable.py#L138-L155
|
|
|
|
Here, we check what args was Ansible constructor was called with.
|
|
|
|
That's pretty much all you need to write safe, rerunnable tests for Press. You
|
|
can checkout https://docs.python.org/3/library/unittest.mock.html for more
|
|
things you can do with the standard python libraries. If your editor and
|
|
plugins are setup configured nicely, you can even do TDD with ease.
|
|
|
|
> Protip: When you have test records you want across a TestCase, then you can
|
|
> simply use the create the test record in `setUp` method of the same. The test
|
|
> records can be assigned to member variables. Eg:
|
|
|
|
```python
|
|
def setUp(self):
|
|
super().setUp()
|
|
|
|
self.team = create_test_team()
|
|
```
|
|
|
|
### Background jobs
|
|
|
|
Since background jobs are forked off of a different process, our mocks and
|
|
patches are not going to hold there. Not only that, but we can't
|
|
control/predict when the background job will run and finish. So, when your code
|
|
involves creating a background job, we can simply mock the call so that it runs
|
|
in foreground instead. There's a utility method you can use to achieve this with ease:
|
|
|
|
http://git.jingrow.com/jingrow/jcloude/blob/23711e2799f2d24dfd7bbe2b6cd148f54f4b253b/press/press/doctype/database_server_mariadb_variable/test_database_server_mariadb_variable.py#L12
|
|
|
|
http://git.jingrow.com/jingrow/jcloude/blob/23711e2799f2d24dfd7bbe2b6cd148f54f4b253b/press/press/doctype/database_server_mariadb_variable/test_database_server_mariadb_variable.py#L104-L108
|
|
|
|
## Running tests
|
|
|
|
You can run all of the tests with the following command.
|
|
|
|
```sh
|
|
bench --site test_site run-tests --app press
|
|
```
|
|
|
|
But you'll never have to. That's what CI is for. Instead, you'll mostly want to use:
|
|
|
|
```sh
|
|
bench --site test_site run-tests --app press --module press.press.doctype.some_doctype.test_some_doctype
|
|
```
|
|
|
|
This is because while writing bugs, your changes will mostly affect that one
|
|
module only and since we don't have many tests to begin with, it won't take
|
|
very long to run a module's test by itself anyway. Give your eyes a break while this happens.
|
|
|
|
You can also run individual test with:
|
|
|
|
```sh
|
|
bench --site test_site run-tests --module press.press.doctype.some_doctype.test_some_doctype --test test_very_specific_thing
|
|
```
|
|
|
|
You most likely won't enjoy running commands manually like this. So you'd want
|
|
to check out [this vim plugin](http://git.jingrow.com/ankush/frappe_test.vim/) or
|
|
[this vscode plugin](https://marketplace.visualstudio.com/items?itemName=AnkushMenat.frappe-test-runner)
|
|
|
|
> Note: frappe_test plugin doesn't populate vim's quickfix list yet. Though
|
|
> Frappe's test runner output isn't very pyunit errorformat friendly, you can
|
|
> still make it work with a [custom errorformat](http://git.jingrow.com/balamurali27/dotfiles/blob/85dc18a/.config/nvim/after/plugin/frappe.vim#LL10C1-L10C128) and some hacks to [set makeprg](http://git.jingrow.com/balamurali27/dotfiles/blob/0bcd6270770d0b67b63fc0ea308e6834fefda5a6/.config/nvim/init.vim#L150C7-L163)
|
|
|
|
# References
|
|
|
|
- https://framework.jingrow.com/docs/v14/user/en/testing
|
|
- https://docs.python.org/3/library/unittest.mock.html
|
|
- https://learnvim.irian.to/basics/compile
|