Commit Graph

52 Commits

Author SHA1 Message Date
Kelvin Zhang
6e0c8c6c37 API Service should return json data directly
Change-Id: Ic5fdefe23443733c3c7e0ae8f351c39117326794
2021-08-31 20:06:57 -07:00
Kelvin Zhang
fd50cb8be1 Rename path to db_path
Change-Id: I32cbd02722a81b54f55df092bc762463df939a9d
2021-08-31 17:28:43 -07:00
Kelvin Zhang
fd33874a47 Ignore vscode files in docker
Change-Id: I464d846c86a6c98c4be2e1993014a799ffc5d7b1
2021-08-31 17:28:42 -07:00
Kelvin Zhang
9b854aa4bf Return error in json
Test: th
Change-Id: Ie9ef8e2bac6447bdc1bf40dcb2b5731b57b47f77
2021-08-31 17:28:42 -07:00
Kelvin Zhang
2ea0bfceb0 Delegate choice of output file path to 1 place in code
Test: th
Change-Id: Ib7b02d5ce4335594b6ca625d850e7176dbdb4f8e
2021-08-31 17:28:42 -07:00
Kelvin Zhang
d7ebc10c0d Cancel polling task after component unmount
Test: th
Change-Id: I0d9058bd24236ea1e22bf91f98b2dbadc83cf8d3
2021-08-31 17:28:42 -07:00
Kelvin Zhang
694daa581f Remove noop try catch statements
Test: th
Change-Id: I5c6fcd5cdd602207e3befc3ff92137d39968c546
2021-08-31 17:28:42 -07:00
Kelvin Zhang
4993e35acf Skip non-zip files when refreshing build library
Test: th
Change-Id: I16631183c40136c4603f20e2ea07664123338da4
2021-08-27 10:53:50 -07:00
Kelvin Zhang
157e46b183 Stop hardcoding a keypath in OTA generator
Test: th
Change-Id: I6f55a36e4559d635bd6cf24fa0b4891ccd2ab9f8
2021-08-27 10:53:23 -07:00
Treehugger Robot
097e8c0a15 Merge "Add support for reuse configuration in OTA generator." 2021-08-19 20:45:53 +00:00
lishutong
1ec526c9a0 Add support for reuse configuration in OTA generator.
Use vuex to store the otaConfig, now the history configurations can be
reused in the job detail page. Also the targetBuilds, incrementalSources
are not stored in vuex as well. Once the targetBuilds are chosen, it
will not lose the states when switching the tab between singleOTA,
batchOTA and chainOTA.

Test: npm run test:unit
Change-Id: I563ac0c79fd0cedd5e71e8722b956545f6624551
2021-08-19 16:34:09 +00:00
lishutong
e8c8d88b8c Fix a bug that may cause the ota generation fail.
The extra space in args['extra'] may cause the ota generation fail.

Test: python test_ota_interface.py -v
Change-Id: Iac6d06a89c52ab7554e54c08e5e04496b3da730a
2021-08-19 15:32:38 +00:00
lishutong
54071681c8 Add test to target_lib.py.
Add test to two classes: BuildInfo and TargetLib. BuildInfo is a
dataclass which can store and parse Android Build informations from an
OTA package. TargetLib is a database interface which can be used to
store and extract BuildInfo.

Please refer to this CL for more details of these two classes:
https://android-review.googlesource.com/c/platform/development/+/1735315

Test: python test_target_lib.py -v
Change-Id: I3b2143af7f3708207b6c53744d903d3dcee92d55
2021-08-17 15:31:28 +00:00
lishutong
70026190c6 Add test to ota_interface.ProcessManagement.
The ProcessManagement helps initiate OTA generation processes and
monitor those processes. Add some test and comments to this class.
Please refer to:
https://android-review.googlesource.com/c/platform/development/+/1736940
for more details.

Test: python test_ota_interface.py -v
Change-Id: Ib22fca4c5a670f8b55db8a4175fef16d92eaceaf
2021-08-12 00:44:42 +00:00
lishutong
1e82f06150 Add test to ota_interface.JobInfo.
Add testcases for ota_interface.JobInfo, which is used as object to
store task information and serve/read data to/from database.
Refer to:
https://android-review.googlesource.com/c/platform/development/+/1736940
for more details.

Test: python test_ota_interface.py -v
Change-Id: I74dcb16390078bd9258da490e21cb2b73dd78e81
2021-08-11 20:45:23 +00:00
lishutong
ec3b42deea Fix a small bug when initialize a ota generation.
The os.path.join() will bring in an unwanted backslash.

Test: Tested by starting a new OTA generation process by calling
ota_interface.ota_generate directly.
Test: python test_ota_interface.py -v

Change-Id: I8449eea79303f5aff5188176538eca1291101dff
2021-08-11 20:24:55 +00:00
lishutong
400253b224 Add jest to dependency.
Jest is a JavaScript test framework. Test-utils is used to test vue
components.

No-Typo-Check: auto-generated artifacts

Test: npm run test.
Change-Id: I2466687120b96a3a393299d127c6d7e1f15204e7
2021-08-11 20:24:55 +00:00
Kelvin Zhang
d689350756 Support docker for ota generator
Test: docker run -it -p 8000:8000 -v target:/app/target -v
output:/app/output zhangxp1998/test:latest

Change-Id: Ife050e6a8c85f2d40aff82a752b19f27c2ab23c5
2021-07-28 22:44:34 -04:00
lishutong
30f991bf6e Fix a bug: redirect to homepage for some url.
In a single page application, the url sometimes does not correspond to
the actual resources on the server. Thus, it might cause 404 when users
refresh the page. By redirecting this kind of url back to the homepage,
the proper page can be rendered. If the url does not exist, the front
end will redirect to a Not Found page.

Test: mannual tested.
Change-Id: I36beff436a450ae7fcabe9172df9c7cc217d7305
2021-07-28 18:14:35 +00:00
lishutong
c4b3dd1aef Fix a bug: multiple flags in ota generator.
Previously, when multiple extra flags are set, the subprocess.run
can start generation properly. Now this is fixed.

Test: mannual tested.
Change-Id: Iab7275b9058a088d1fafd8c445c7f0775626c1ba
2021-07-28 17:51:31 +00:00
lishutong
bb3bc961d4 Add support for chain OTA generation.
To upgrade from the Android version A to Android version D, one could
directly generate an OTA package from A to D. But chances are (a) this
OTA package can be large and unstable (b) there are multiple other
devices are on version B or C. So generation of chain OTA packages
(A-->B-->C-->D) can help life easier.

Users will be able to select and sort the Android build in
`ChainOTAOptions.vue` component, and submit multiple jobs at the same
time using a OTAConfiguration from `JobSubmission.js`.

Test: Mannual tested.
Change-Id: I9f16f981af80900c18a571162146ce218ea96387
2021-07-23 19:52:28 +00:00
lishutong
c4b6c3a7dd Add batch generation to OTAGUI.
Batch generation of OTA packages is a important feature
requested by googler and partners: please refer to go/ota-dashboard-doc.
Given n incremental source builds and m target builds,
batch generation will generate n x m OTA packages in total.
If n=0, full OTA package will be generated.

The front end will be taking in the source/target lists and send the
request to backend one-by-one.

Test: mannual tested.

Change-Id: I769359ee69c7aa8c71704c4e119c374635554dfb
2021-07-23 19:18:52 +00:00
lishutong
aaf2c7a933 Refactor the frontend for batch generation.
Following part has been modified:

1. Add tabs for selection between single generation and batch
generation. (src/components/JobConfigure.vue)

2. Change the data structure of OTAConfiguration, now it only records
the flags. The source/target build will be provided when submit jobs.
(src/services/JobSubmission.js)

3. Seperate the OTAOptions as a single component, which only takes in
the flags for backend. The selection of source/target build will be in a
seperate component. (src/components/OTAOptions.vue,
src/components/SingleOTAOptions.vue).

4. Now the partition selection can takes in more than one build, but
only show the partition list of first one. Later on, this will be able
to show the intersection of the partition lists from all given builds.
(src/components/PartialCheckbox.vue)

Point 1 enables the possibility of the dynamical loading of single/batch
ota generation pages. Point 2,3,4 allow the OTAOptions components to
be reused for batch generation.

Test: Mannual tested.
Change-Id: I1a29fa7c605596d717d19da25d31b81ce5b8fcba
2021-07-23 15:09:31 +00:00
lishutong
f084110dbe Fix a bug that job might not be able to be started properly.
A variable name was mispelled and it could lead to jobs unable to be
started.

Test: Mannual tested.
Change-Id: I4067e2ae243428cb190463f55122b88d471f45f7
2021-07-20 18:27:07 +00:00
lishutong
fc0431fda1 Add all possible flags to the front end.
A complete set of flags for the CLI tool can be found in this script:
https://source.corp.google.com/android/build/make/tools/releasetools/ota_from_target_files.py

Now most of the flags are being added to the front end. However, there
are several TODOs:
1. flags that require extra arguements or file operations are not
supported yet.
2. the flags can depend or exclude on other flags. Dependency is
included in file 'JobSubmissions.js'-'OTAExtraFlags'. But the function
is not implemented yet.

Test: Mannual tested.
Change-Id: I3b6b76d48861cff81bfad86b549ff53f6536933b
2021-07-20 15:31:47 +00:00
lishutong
b9d9f55657 Fix a bug in build library: the uploaded file cannot be shown in table.
Previously, the newly uploaded file cannot be shown immediately in the
datatable, unless refresh the page. Now the bug is fixed.

Similar bug is also fixed in OTAJobTable.vue.

Test: mannual tested.
Change-Id: Ia0531db884a7e1675b2cf51aeb8ca0847694c63c
2021-07-19 20:45:46 +00:00
lishutong
029dbe681e Move the '--partial' flag and output configuration to the backend.
Previously, the front end will generate a string for '--partial' flag.
Now the front end will only record the partial partitions that are going
to be updated as an array, and the backend will convert the array into a
string for CLI tool.

The output now is also calculated in the backend,

Test: mannual tested.
Change-Id: Ib5604e52b0c0d0cdd2bf85c71479b3133479da74
2021-07-19 17:57:07 +00:00
lishutong
1ab3abb53a Refactor the OTA configuration page using several components.
The original code for OTA configuration in the frontend is tightly
coupled and hard to maintain. This CL decoupled the job configuration
component (OTAOptions.vue), build library (BuildLibrary.vue) and job
submission (JobSubmission.js).

The BuildLibrary.vue will present the
target build list and take selections. Selections will be sent to
OTAOptions.vue, where flags are set. Once the setting is complete, an
OTAOption object defined by JobSubmission.js will be sent to the
backend. This design can be viewed in go/ota-dashboard-doc : detailed
design - TASK CONFIGURATION.

This has several benefits: (a) easier for
future maintainence. (b) The original code cannot easily be reused for
batch OTA package generation. Now, only the OTAOptions.vue has to be
modified for this purpose.

Test: Mannual tested. Unit tests will be added in following CLs.

Change-Id: I846d0c242cd6ed51478dd4d1a3e4c5fb8878aba1
2021-07-19 16:22:41 +00:00
lishutong
be609f5b31 Add responsive design to the UI.
Now the columns width can be adjusted according to the window size.

Test: Mannual tested.
Change-Id: I1e2911b757948ee95f487631bcdceb5ede1fa63e
2021-07-16 19:38:16 +00:00
lishutong
c52791ae38 Add data table for build library.
Currently, users cannot choose build from the datatable directly, but
the datatable will give options to sort the builds uploaded and then
select from the sorted list.

Test: mannual tested.
Change-Id: If4e8ff7210a19170f7a7e6f77f03d21f6a10e6ac
2021-07-16 18:02:14 +00:00
lishutong
5ad5ba9167 Fix few UI bugs.
1. The ordering in OTAJobTable was incorrect. Now fixed.
2. The JobDisplay card in JobList wasn't rendered correctly.

Test: Mannual tested.
Change-Id: I53d861fb60a623552487f957b42f4cd7404e654e
2021-07-15 15:56:43 +00:00
lishutong
2ad7328834 Add data table in OTAGUI.
Use data table for build library and task monitoring.

Test: Mannual tested.
Change-Id: Icefd6bd27c98fe6b213ade7239ad4918fa9a9485
2021-07-14 17:41:53 +00:00
lishutong
0bb19984b5 Add dependency for data table.
Add vue3-table-lite.

No-Typo-Check: auto-generated artifacts

Test: Mannual tested.
Change-Id: Ifc84c61861f88f9eeb8d3e4d6775c93aaa609318
2021-07-13 15:32:46 +00:00
lishutong
5781604572 Use material design for OTAGUI.
1. Changed the layout using material design ui in vuetify.
2. Changed the file upload to drag-and-drop style.

Test: Mannual tested.
Change-Id: I8b34a405ae3783d6806946f0adc5fc5ac9c30a49
2021-07-13 15:32:44 +00:00
lishutong
d5c8b3a43c Add dependency for vuetify (material design UI).
No-Typo-Check: auto-generated artifacts

Test: Mannual tested.
Change-Id: Ia73480d3bf0617f4089f2da1a1f9d7ec13de42bb
2021-07-13 15:23:55 +00:00
lishutong
9a5b57e68d Seperate the ota_analysis from OTAgui.
The default entry point is /analyseOTA in production enviroment. This is
for the deployment on android.github.io/.

Test: Mannual tested.
Change-Id: Ic77277024b34b67b9964be8cf4f1592cebf5c5e8
2021-07-08 16:05:13 +00:00
lishutong
52a5bf8b52 Add some comments for a consistent javascript style.
Follow the requirement of:
https://google.github.io/styleguide/jsguide.html#jsdoc-top-file-level-comments

Test: Mannual tested.
Change-Id: I697879384f64120a0effda00e70ec1b20260746b
2021-07-01 21:13:07 +00:00
lishutong
b7e6e67a8a Add support for analysing the COW merge operations in OTA packages.
Now the OTA_analysis is able to do statistics over how many blocks
are being operated by different types of COW merge operations.
Also refactor part of payload.js.

Test: Mannual tested.
Change-Id: Icd24f2c0aa42f968438965378b05a7021bafb88c
2021-07-01 20:49:11 +00:00
lishutong
37573fbca6 Update to new update_metadata.proto which support COW merge operation.
Now update_metadata.js will be able to parse COW merge operations in the
OTA packages. Please refer to change 1736935 for more details.

No-Typo-Check: auto-generated artifacts

Test: Mannual tested.
Change-Id: Ie18125524b991173b0599704eca51004fd0c0243
2021-07-01 18:32:01 +00:00
lishutong
18e0c887d8 Add 'Select/Unselect All' in partition selection.
Now both OTA configuration and OTA analysis page support select/unselect
all partitions.

Test: Mannual tested.
Change-Id: I684912886f3d3da259aeafea6aea1e53ae4a1c9a
2021-06-30 22:02:51 +00:00
lishutong
8a609acf15 Use pie chart for visualization of anaylsis result.
echarts_data.js will take the raw data and convert it into vue-echarts
readable option. PieChart.vue take this option and render it into a
pie chart.

Test: Mannual tested.
Change-Id: I54af3603031f0e23afa743ffc420d240a1b07bb2
2021-06-30 21:34:38 +00:00
lishutong
5a782225f0 Add functions to analyse the composition of payload.
payload_composition.js defines functions that could parse the
manifest of payload and do statistical analysis based on different
metrics. Currently, there are two functions:
1. Number of blocks (in target build) that are being operated,
categorized by the installation operations.

2. Disk usage of the payload.bin, categorized by the installation
operations.

The output is currently a list of pairs: (Operation, Number), which
can be later turned into input of visualized element.

Test: Mannual Tested.
Change-Id: I07defc23f6f04616656d8c9d3a7ecd05026bbbff
2021-06-30 21:31:46 +00:00
lishutong
ff63753f53 Add support for analysis of OTA package.
payload.js defines a Class that could parse the OTA metadata from
an OTA package(.zip). PackageAnalysis.vue enable users to upload
their own OTA package for analysis. PayloadDetail.vue and
OperationDetail will display the metadata in a human-readable way.

The whole analysis process is done on frontend and no communication
with server is involved after loading the page.

Test: Mannual tested.
Change-Id: I3becba872f6564ec9e44edad3a6cc19eb0714ed6
2021-06-28 19:19:49 +00:00
lishutong
76e4550e84 Add the compiled 'update_metadata.proto' file.
Use protobufjs to compile the update_metadata.proto
into update_metadata_pb.js

No-Typo-Check: auto-generated artifacts

Test: Mannual tested.
Change-Id: Ia0fefa57154f68809c8775ead9fa5e24d4804013
2021-06-26 15:55:51 +00:00
lishutong
d16272de68 Add the dependency of echarts.
No-Typo-Check: auto-generated artifacts

Test: Mannual tested.
Change-Id: I1d29383a8b26d73a1c4075c5e7a2516ec3407d8d
2021-06-25 22:47:56 +00:00
lishutong
c2c0ed1f6e Use database to store ota generate history.
Add functions:
- Now the ota package generate history is stored in a database.
- Job status page will show some basic info of the jobs.
- Job detail page will show a detailed configuration.

Test: Mannual tested.
Change-Id: Ia98763e783e8d9c903ac8f5c137719df8527d204
2021-06-21 19:17:34 +00:00
lishutong
71cab6686b Add an alert in frontend when job cannot be submitted.
Test: Mannual tested.
Change-Id: I9076be48cf2efd8d545b88beb87ac0db651d39e4
2021-06-15 14:07:05 +00:00
lishutong
dce3afb0e9 Add support for partial update and target library.
New functions:
- The uploaded target files will be stored into a database.
- The detailed info of target files can be shown in the target library.
- Add support for generating partial OTA.

Test: Mannual tested.
Change-Id: Idf715994fe51b9ab6addbfe72792be890243702d
2021-06-15 13:59:57 +00:00
lishutong
fcdf777e5e Add support for uploading and downloading.
New functions:
- Can upload target file.
- Can select target file from the uploaded ones.
- Can download the generated OTA file.
- Can monitor the stdout during OTA package generation.

Also decoupled the job running function with the web server code.

Test: Mannual tested.
Change-Id: Ic525ef3997f048d73b21ad1d9475d54b1f8d112c
2021-06-08 16:54:40 +00:00
lishutong
f0e53f58aa Add some configuration files.
Test: Mannual tested.
Change-Id: I688bf65cadfc094800635b1e26b51411b473bb6a
2021-06-07 03:12:46 +00:00