Dynamic On-Demand Crowdshipping Using Heuristics-Embedded and Constrained Deep Reinforcement Learning

Crowdsourced parcel delivery, or crowdshipping, has emerged recently as a crucial component of city logistics and an attractive alternative to conventional van/truck-based delivery. By employing ordinary people (termed crowdsources) who use spare time to perform pickup and delivery by walking, biking, or driving personal cars as opposed to using company-owned vans or trucks, crowdshipping helps reduce shipping cost and freight traffic-related congestion, parking demand, energy use, and emissions, all of which contribute to more sustainable urban communities. In this talk, I will present my group’s recent research in heuristics-embedded and constrained deep reinforcement learning (DRL) to tackle the key problem of crowdsource-shipping request assignment in a dynamic on-demand crowdshipping environment. The idea of heuristics-embedded training is conceived by designing an elaborate action space where refined local search heuristics are conceived and embedded to direct the specific action to take once an action type is chosen by DRL, to preserve tractability of DRL training. To tackle the hard constraints pertaining to crowdsource and shipping request time windows, we propose and integrate three new strategies (feasibility enforced local search, multiple schedules with different penalties, and exponential penalty) as part of the DRL training and testing. Extensive numerical analysis on adopting the proposed approach with multiple DRL algorithms is performed. By comparing the proposed approach with conventional heuristic methods, benchmarking the results against global optimality, and testing result sensitivity to demand pattern variations, weD demonstrate superior performance of the approach in solution quality, computational efficiency, and robustness, and thus potential for practical implementation.