/usr/local/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp WARNING 2022-07-26 10:44:12,183 launch.py:423] Not found distinct arguments and compiled with cuda or xpu. Default use collective mode INFO 2022-07-26 10:44:12,183 launch_utils.py:641] Change selected_gpus into reletive values. --ips:0,1,2,3 will change into relative_ips:[0, 1, 2, 3] according to your CUDA_VISIBLE_DEVICES:['0', '1', '2', '3'] INFO 2022-07-26 10:44:12,185 launch_utils.py:528] Local start 4 processes. First process distributed environment info (Only For Debug): +=======================================================================================+ | Distributed Envs Value | +---------------------------------------------------------------------------------------+ | PADDLE_TRAINER_ID 0 | | PADDLE_CURRENT_ENDPOINT 127.0.0.1:60001 | | PADDLE_TRAINERS_NUM 4 | | PADDLE_TRAINER_ENDPOINTS ... 0.1:60002,127.0.0.1:60003,127.0.0.1:60004| | PADDLE_RANK_IN_NODE 0 | | PADDLE_LOCAL_DEVICE_IDS 0 | | PADDLE_WORLD_DEVICE_IDS 0,1,2,3 | | FLAGS_selected_gpus 0 | | FLAGS_selected_accelerators 0 | +=======================================================================================+ INFO 2022-07-26 10:44:12,185 launch_utils.py:532] details abouts PADDLE_TRAINER_ENDPOINTS can be found in log/endpoints.log, and detail running logs maybe found in log/workerlog.0 ----------- Configuration Arguments ----------- backend: auto elastic_server: None force: False gpus: 0,1,2,3 heter_devices: heter_worker_num: None heter_workers: host: None http_port: None ips: 127.0.0.1 job_id: None log_dir: log np: None nproc_per_node: None run_mode: None scale: 0 server_num: None servers: training_script: tools/train.py training_script_args: ['-c', 'configs/picodet/picodetv2_npu_s_416_coco.yml', '--eval'] worker_num: None workers: ------------------------------------------------ launch train in GPU mode! launch proc_id:10788 idx:0 launch proc_id:10791 idx:1 launch proc_id:10794 idx:2 launch proc_id:10797 idx:3 /usr/local/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly server not ready, wait 3 sec to retry... not ready endpoints:['127.0.0.1:60003'] I0726 10:44:18.165164 10788 nccl_context.cc:74] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0 W0726 10:44:18.838285 10788 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 6.1, Driver API Version: 10.2, Runtime API Version: 10.1 W0726 10:44:18.843362 10788 device_context.cc:465] device: 0, cuDNN Version: 7.6. loading annotations into memory... Done (t=20.74s) creating index... index created! [07/26 10:44:51] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 200365, area: 0.0 x1: 296.65, y1: 388.33, x2: 297.67999999999995, y2: 388.33. [07/26 10:45:08] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 550395, area: 0.0 x1: 9.98, y1: 188.56, x2: 15.52, y2: 188.56. [07/26 10:45:14] ppdet.utils.download INFO: Downloading PPLCNet_x0_75_pretrained.pdparams from https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/legendary_models/PPLCNet_x0_75_pretrained.pdparams 0%| | 0/9252 [00:00