In this article, we investigate efficient transmission of Object-based MPEG-4 video over IP networks with QoS management capabilities. MPEG-4 Audio Visual Objects (AVOs) are classified based on application-level QoS criteria and AVOs semantic descriptors according to MPEG-7 framework. MPEG-4 AVOs requiring same QoS performance from the network are automatically classified and multiplexed within one of the IP DiffServ PHB (Per Hop Behaviors). Object data-packets within the same class are then transmitted over the selected transport layer with the corresponding bearer capability and priority level. We propose to extend the MPEG-4 System architecture with a new "Media QoS Classification Layer". This layer provides automatic and accurate mapping between MPEG-4 application-level QoS metrics and underlying transport network with QoS mechanisms such as IP DiffServ. The "Media QoS Classification Layer" makes use of a neural network classification model that is transparent to application and network layers. Implementation and performance evaluation of the proposal are also described.