# DarkWebID Organization Provisioning Tool

A standalone Python script that bulk-provisions organizations via the
**DarkWebID V2 External API** using your API credentials.

---

## Prerequisites

| Requirement | Notes |
|-------------|-------|
| Python 3.9+ | `python --version` |
| `requests` library | `pip install requests` |
| `openpyxl` library | `pip install openpyxl` |
| API credentials | Email + password with **`Permit access to web services`** role and **`partner administrator`** or **`partner agent`** role |
| IP whitelist | Your machine's IP must be whitelisted in your DarkWebID |

Install both libraries at once:

```bash
pip install requests openpyxl
```

---

## Quick Start

1. **Install dependencies** (one-time):
   ```
   pip install requests openpyxl
   ```

2. **Prepare your data file** — copy `provisioning_data_template.xlsx` to any folder you like, fill in your Organization rows (one row per Organization), and delete the example row.

3. **Dry-run first** — validates your file without touching the API:
   ```
   python provision_organizations.py --url https://DARKWEBID_URL/ --username {darkweb_api_user_email_address} --password {darkweb_api_user_password} --file provisioning_data.xlsx --dry-run
   ```

4. **Run for real:**
   ```
   python provision_organizations.py --url https://DARKWEBID_URL/ --username {darkweb_api_user_email_address} --password {darkweb_api_user_password} --file provisioning_data.xlsx
   ```

5. **Check the results** — two files are written to the **same folder as your data file**:
   - `provisioning_results_YYYYMMDD_HHMMSS.xlsx` — colour-coded per-step status, updated after every row
   - `provisioning_log_YYYYMMDD_HHMMSS.txt` — full console transcript including every request body, HTTP status, and timing

---

## Running the Script

### File locations

The script and your data file can be in **any folder** — there is no fixed location requirement.
The `--file` argument accepts an absolute path or a path relative to your current directory.
The results file is written to the **same directory as `--file`** unless you specify `--output`.

A typical layout:

```
C:\provisioning\
    provision_organizations.py
    provisioning_data.xlsx        ← your filled-in data file
    provisioning_results_*.xlsx   ← generated by the script
```

### Windows (Command Prompt or PowerShell)

```
python provision_organizations.py ^
    --url      https://DARKWEBID_URL/ ^
    --username {darkweb_api_user_email_address} ^
    --password {darkweb_api_user_password} ^
    --file     provisioning_data.xlsx
```

PowerShell uses a backtick for line continuation:

```powershell
python provision_organizations.py `
    --url      https://DARKWEBID_URL/ `
    --username {darkweb_api_user_email_address} `
    --password {darkweb_api_user_password} `
    --file     provisioning_data.xlsx
```

### macOS / Linux

```bash
python3 provision_organizations.py \
    --url      https://DARKWEBID_URL/ \
    --username {darkweb_api_user_email_address} \
    --password {darkweb_api_user_password} \
    --file     provisioning_data.xlsx
```

### Common variants

```bash
# Validate + preview — no API calls made
python provision_organizations.py ... --file data.xlsx --dry-run

# Save results to a specific file
python provision_organizations.py ... --file data.xlsx --output C:\Reports\results.xlsx

# Use a CSV file instead of Excel
python provision_organizations.py ... --file data.csv

# Show all available options
python provision_organizations.py --help
```

---

## Command-line Reference

```
python provision_organizations.py [options]

Required:
  --url       URL      Base URL of DarkWebID (include trailing slash).
                       Example: https://DARKWEBID_URL/ (include trailing slash)
  --username  EMAIL    Your API username (email address).
  --password  PASS     Your API password.
  --file      PATH     Path to provisioning data file (.xlsx or .csv).

Optional:
  --output    PATH     Path for the results file.
                       Default: provisioning_results_YYYYMMDD_HHMMSS.xlsx
                                (same directory as --file)
  --delay     SECONDS  Pause between Organizations to avoid API rate-limiting (default: 2.0).
                       Set to 0 to disable. Increase if you see connection timeouts
                       on Step 1 after several consecutive successful Organizations.
  --dry-run            Validate the file and print the plan. No API calls.
  --help               Show this help message and exit.
```

---

## Input File

The script accepts **`.xlsx`** and **`.csv`** formats.  One row = one Organization.

Start with `provisioning_data_template.xlsx` — it contains all column headers,
an example row, and a notes row explaining each field.

### Column reference

Column headers are matched **case-insensitively** and stripped of surrounding
whitespace, so `Organization Title`, `Organization title`, and `  Organization title  ` all work.

#### Organization Identity

| Column | Required | Accepted values |
|--------|----------|-----------------|
| `Organization Title` | **Yes** | Any string, max 128 chars |
| `Industry` | **Yes** | See [Industry values](#industry-values) below |
| `Employee Count` | **Yes** | `1-10`, `11-50`, `51-100`, `101-250`, `251+` |

#### Domain Monitoring

| Column | Required | Notes |
|--------|----------|-------|
| `Domain` | **Yes** | Real registered FQDN, e.g. `example-company.com`. The API validates domains against live DNS — synthetic or non-existent domains will be rejected. Mandatory: if domain setup fails the script skips user creation and all later steps for that row. |

#### User

| Column              | Required | Notes                                  |
|---------------------|----------|----------------------------------------|
| `User Email`        | **Yes**  | Must be globally unique                |
| `User First Name`   | **Yes**  |                                        |
| `User Last Name`    | **Yes**  |                                        |
| `User Account Type` | No       | `Standard` (default) or `Privileged`   |

#### Organization-level Notifications

| Column | Required | Notes |
|--------|----------|-------|
| `Notification Preference` | No | `Never` (default), `Monthly`, `Daily and Monthly` |
| `Notification Date Format` | Conditional | `M/D/Y` (default), `D-M-Y`. **Required when Preference ≠ Never** |
| `Notification Emails` | Conditional | Comma-separated. **Required when Preference ≠ Never** |
| `Clean Bill of Health` | No | `Yes` or `No` (default) |
| `CBoH Emails` | Conditional | Comma-separated. **Required when CBoH = Yes** |
| `CBoH Reply-To` | No | Single email address |

#### User-level Preferences

| Column | Required | Notes |
|--------|----------|-------|
| `User Notification Preference` | No | `Never` (default), `Monthly`, `Daily and Monthly` |
| `User Date Format` | No | `M/D/Y` (default), `D-M-Y` |

#### Reporting Preferences

| Column | Required | Notes |
|--------|----------|-------|
| `Report Date Format` | No | `M/D/Y` (default), `D-M-Y` |
| `Report Executive Summary` | No | `Yes` or `No` (default). **Must be `Yes` if any sub-section below is `Yes`** |
| `Report Benchmark Averages` | No | `Yes` or `No` (default) |
| `Report Monitoring` | No | `Yes` or `No` (default) |
| `Report Organizational Compromises` | No | `Yes` or `No` (default) |
| `Report Breaches` | No | `Yes` or `No` (default) |
| `Report Dark Web Benefits` | No | `Yes` or `No` (default) |

#### Automated Reports

| Column | Required | Notes                                                                                                                                       |
|--------|----------|---------------------------------------------------------------------------------------------------------------------------------------------|
| `Automated Report` | No | `Yes` or `No` (default). Master toggle — set to `Yes` to enable automated reports for this Organization                                              |
| `Monthly Business Report` | Conditional | `Yes` or `No` (default). Send a monthly automated report. **At least one of Monthly/Quarterly must be `Yes` when Automated Report = `Yes`** |
| `Quarterly Business Report` | Conditional | `Yes` or `No` (default). Send a quarterly automated report. Both columns may be `Yes` simultaneously                                        |
| `Automated Report Recipients` | Conditional | Comma-separated. **Required when Automated Report = Yes**                                                                                   |
| `Automated Report Subject` | Conditional | **Required when Automated Report = Yes.** Supports template tags: `[report_type]`, `[start_date]`, `[end_date]`, `[link]`                   |
| `Automated Report Body` | Conditional | Plain text — the script Base64-encodes it automatically. Supports template tags: `[report_type]`, `[start_date]`, `[end_date]`, `[link]`                                                                                   |

---

### Industry values

| Value | Value | Value |
|-------|-------|-------|
| Aerospace & Defense | Finance & Insurance | Non-Profit Organization |
| Business & Professional Services | High-Tech & IT | Pharmaceutical |
| Construction & Engineering | Hospitality | Retail & eCommerce |
| Education & Research | Legal | Service Provider |
| Energy & Transportation | Manufacturing | State/Local Government |
| Federal Government | Media & Entertainment | Systems Integrator |
| | Medical & Healthcare | Wireless Industry |
| | | Other |

Values are matched **case-insensitively**, so `Other`, `other`, and `OTHER` all work.

---

## How the Script Works

For each row the script runs up to **7 sequential API steps**:

| Step | API call | Description |
|------|----------|-------------|
| 1 | `POST /services/v2/organization` | Create the Organization |
| 2 | `PUT  /services/v2/organization/notification/{id}` | Set notification preferences |
| 3 | `PUT  /services/v2/organization/reporting-preferences/{id}` | Set report content preferences |
| 4 | `POST /services/v2/monitor/{id}/domain` | Add the monitored domain |
| 5 | `POST /services/v2/user` | Create the user |
| 6 | `PUT  /services/v2/user/{id}/preferences` | Set user preferences |
| 7 | `PUT  /services/v2/reporting/{id}` | Configure automated report *(skipped when Automated Report = No)* |

### Fail-fast behavior

If **any step fails** — including Step 1 — all subsequent steps for that Organization
are immediately marked **SKIPPED** and the script moves to the next row.

```
[  1/98 |   0%]  Example Company

  Step 1 Create Organization
    POST  /services/v2/organization
    body: {"title":"Example Company","industry":3,"employee_count":1}
    --> 201  id=12345  (0.43s)

  Step 2 Notifications
    PUT   /services/v2/organization/notification/12345
    body: {"notification_preference":1,"clean_bill_email":false,...}
    --> 200  (0.21s)

  Step 3 Reporting Prefs
    PUT   /services/v2/organization/reporting-preferences/12345
    body: {"date_format":0,"dw_executive_summary":false,...}
    --> 200  (0.18s)

  Step 4 Domain
    POST  /services/v2/monitor/12345/domain
    body: {"domain":"example-company.com"}
    --> 422  FAILED  (0.31s)
    error: {"errors":{"domain":["already monitored by another organization"]}}

  Step 5 Create User  [SKIPPED]
  Step 6 User Prefs   [SKIPPED]
  Step 7 Automated Report  [SKIPPED]

  --> FAILED  (1.1s)
```

A failed row does **not** stop the script — provisioning continues for the
remaining rows.

### Validation before any API calls

The script validates **all rows** before making a single API call. Every
problem in every row is reported upfront so you can fix the whole file in
one go rather than discovering errors one at a time.

```
[VALIDATION FAILED] 3 error(s):

  - Row 2: 'Domain' is required but missing or empty.
  - Row 3: Unknown Industry 'Finance'. Valid values: "banking & finance", ...
  - Row 3: 'Notification Emails' is required when Notification Preference is 'Monthly'.
```

---

## Output Files

Every real provisioning run (not `--dry-run`) writes **two files** to the
same directory, both stamped with the same timestamp:

```
provisioning_results_20260812_143000.xlsx
provisioning_log_20260812_143000.txt
```

### Results spreadsheet (`.xlsx`)

Updated after every row — **partial results are preserved** even if the script
is interrupted mid-run.

#### Columns

| Column | Description |
|--------|-------------|
| `#` | Row number from the input file |
| `Organization Title` | Name of the Organization |
| `Organization ID` | Numeric ID assigned by the API (filled when Step 1 succeeds) |
| `Step 1` – `Step 7` | `OK`, `FAIL`, `SKIPPED`, or `N/A` |
| `User ID` | Numeric ID of the created user |
| `Overall Status` | `SUCCESS` or `FAILED` |
| `Errors` | All error messages for this row, one per line |

#### Cell colour coding

| Colour | Meaning |
|--------|---------|
| Green | Step completed successfully (OK) |
| Red | Step failed |
| Yellow | Step skipped due to a prior failure |
| Grey | Step not applicable (N/A) |

### Log file (`.txt`)

A plain-text transcript of everything printed to the console, written
**line-by-line** (line-buffered) so it remains fully readable even if the
process is killed or crashes mid-run. Use this file to diagnose exactly what
happened: you can see every request body, every HTTP status code and response
time, every retry attempt, and the point at which provisioning stopped.

The log file contains:
- **Run header** — file paths, target URL, Organization count, start time
- **Per-Organization banner** — `[N/total | %]  Organization Title`
- **Per-step detail** — HTTP method, endpoint path, request body (JSON), response status and elapsed time
- **Error responses** — full response body on any non-2xx reply
- **Retry lines** — each retry attempt for Step 6 (user sync delay)
- **Progress bar** — after every Organization: `[=====>  ] N/total (%) elapsed ETA`
- **Final summary** — succeeded / failed counts, total elapsed time

The log file path is printed in both the run header and the final summary so
you always know where to find it.

---

## CSV Input

The script accepts `.csv` files with the same column headers as the template.
UTF-8 encoding is expected; a BOM (``) is handled automatically.

To export the template as CSV: **File → Save As → CSV UTF-8 (Comma delimited)**.

---

## Dry Run

Use `--dry-run` to validate the file and preview what would be provisioned
without making any API calls:

```bash
python provision_organizations.py \
    --url https://DARKWEBID_URL/ \
    --username {darkweb_api_user_email_address} \
    --password {darkweb_api_user_password} \
    --file provisioning_data.xlsx \
    --dry-run
```

---

## Authentication

The script uses **HTTP Basic Authentication** (Base64-encoded `username:password`).
API access is secured by IP address whitelisting instead of 2FA.

The authenticated user must have:
- the `Permit access to web services` enabled, **and**
- the `partner administrator` or `partner agent` role

Your machine's IP address must be setup for the user on the IP whitelist for web services.

---

## Troubleshooting

| Error | Likely cause                                                                                                                            | Fix                                                                                                                                      |
|-------|-----------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
| `HTTP 401 Invalid credentials` | Wrong username or password                                                                                                              | Re-check `--username` / `--password`                                                                                                     |
| `HTTP 403 IP address not in whitelist` | Your IP is not whitelisted                                                                                                              | Add your IP address in your user settings                                                                                      |
| `HTTP 403 User does not have the required 'Permit access to web services'` | Account not set up for API access                                                                                                       | Enable the `Permit access to web services` to the account in your partner portal                                                         |
| `HTTP 422` on Step 1 | Organization title already exists, or invalid field values                                                                              | Check the `Errors` column in the results file                                                                                            |
| `HTTP 422` on Step 4 — "not a valid domain" | The API validates provided domains — synthetic names or invalid domains are rejected                                                    | Use the Organization's real registered domain                                                                                                     |
| `HTTP 422` on Step 4 — "already monitored" | Domain is already being monitored by another organization                                                                               | Use a different domain, or remove the existing one first                                                                                 |
| `HTTP 422` on Step 5 | User email already in use globally                                                                                                      | Use a unique email address                                                                                                               |
| `HTTP 422` on Step 7 — "The subject field is required" | `Automated Report Subject` cell is blank. The script validates this upfront; if you bypassed validation this is the API's own rejection | Fill in `Automated Report Subject` for every row where `Automated Report = Yes`. The field is required — there is no server-side default |
| `HTTP 404` on Step 6 — all retries exhausted | User record sync took longer than 20 s (5 attempts × 5 s gap)                                                                           | Increase `--delay` to give more time between Organizations; the Organization itself was created — only the user prefs are missing                          |
| Connection timeout on Step 1 after several successful Organizations | API rate-limiter triggered by rapid successive requests                                                                                 | Use `--delay 5` (or higher) to slow the provisioning rate                                                                                |
| Validation failed before any API calls | Missing required columns or invalid values                                                                                              | Read the error list printed to the console                                                                                               |
| Need to debug a mid-run failure | Process crashed or was killed                                                                                                           | Open `provisioning_log_*.txt` — it is written line-by-line so it is fully intact up to the last completed step                           |